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AESIBACT 

^ Besea^rch was conducted to determine how well the 

Management Audit Survey (MAS) , an^ evaluation system us^ed in the O.S. 
EmploymeVt service (ES) , could predict various ES performance; 
critefia7\(Th^P-uxpose of the system is to give- all levels of 
management^ an^nalysis ^nd review of management procedures and of 
certain organizational climate dh&racteristics .to stimulate higher, 
deyels of performance through better, use of hum^ resources,) Effects 
of management' practices <?n productiyity of ES offices were examined 
through empldyee responses to the lOD-item ,MAS questionnaire, Ihe 
primary statistical analyses were done' on 265 offices from six ^' 
States. Fifteen criterion measures and fifteen control scores weM 
examined to assess productivity of the ES system. 5ix criterion 
measures were identified as being most relevant for an-alyzing ES 
performance. Two control score:?^ — workload (applicants available/ES 
p-ositions) and 01 clj^imants as a -percentage or applicants v 
available — explained fhe most variance on selected, criteria. Workload 
had a positive impact on .placement per position and a' negative impact 
on percent of applicants placed, Dnreiial)ility of criterion measures 
and effeciis of control variables maid^ predictionr^of ES performance 
criteria with the MAS scfbtes difficult. The MAS scores with the most 
signifiqant- validities , in predicting ES perf ormance ^criteria were 
operational, efficieiicy^ performance^; feedback^ work satisfaction/ 
morale^ and satisfaction with pay. Study results? indicated that 
further research is needed^. but that management practices assessed by^ 
the MAS plaV a signif;icajit role in the productivity o^ the Employment 
rvica. In5:lud€d-^n the report are background ^ce^airch reviews oJi • ' 
ganiisational climate and effectiveness and'a description .of - ^ 
previous research, of the MAS. (Author/HL) 
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CHAPTER i 
/ ' INTRODUCTION 



The United States Employment Service first began in 1933, with the passage . 
of the Wagner-Peyser Act. The primary role of the Employment -Servibe was 
job plaqement- -matching individuals to available positions thereby serving 
both the unemployed and employers. Over the years, other legislation has , 
mandated other responsibiliftiW and duties for the Employment Service. The 
Employment Service was res^lpnsi^le for much of the training of «the occupa-- 
tionally. unfit for a period of time, and was' responsible for delivering man 
power services to tfie disadvantaged (the poor, minority group members, 
handicappedi the young, the old, etc.) for another period. In Addition to i 
number of other functions, the- emphasis has again: shifted back to placement^ 
with ^increasing emphasis on quality. . • , . , • 

Since the Employment Service is politically-based and ^epends upon the pej- 
ceived needs of society "for its goals and objectives ^he fi^citions of that 
agency will continue to change from time to,. time,| with different ejnphases 
according to the current social conscience and the curi^.ent i^litical exigen 
cies. The difficulty with any legislated piiblic kgency was pointed'out by 

Johnspn (1973) : . ' " ' .-^ ' 

', ' <i • 

.Between the conceptualization of a public service an^ its 
final implementation lies a tortuous and offdri devious path, yt - 
begins with what the courts call the "intentM of the enabling \- . / 
legislation, winds through the verbiage of thfeimandate issued by^ - 
agency policy makers, drifts down, through the H^rafchy 6f yar- 
. ious administrati;kre l^vel^and finally reaches the front lines- - 
that point at which" sterviceis delivered to the public. The^pro- ^ 
cess is the same^ wheHer the initial intent |5 to provide educa- . 
tion, to adminisj:©tvwel]fare, to ■ ensure civil rights, or, to deal -it,: 
' with public offenders: A government •bureau«:racy with responsi- 
bility for the particular function must devise a delivery sysr 
tem, presiimably to provide a means for the program to , find and . 
serve its intended target group. . 

The consequences of this process are- manifold. Often poili-^ 
'cies enunciated by high Jlev^l agency spokesmen are -altered, ' , 
diluted, rigidified, or reinterpreted as they movtf toward the ^ . 
point of implementation. Ijhus tTie service "finally delivered at 
• the lowest point of the structure may^bear little resemblance 
to the ideals 'stated in pubUic^pronm^i^eihents of policy makers 

and administrator^ . . — \ / \l_ • , - • 

Manpower programs are Mttle/dif ferent from other social. , 
programs in this re^^rd. ■ NoNu^er what the original irftent, 
no matter how tortuous the path from conception to implementav 
tion, it is in- the local offices of the ppblic- employnfent ser-\ 
vice throughout the country .th^t most manpower service.s^ finally 
' ci^ss over to the recipient.. It is at that point where mter- 
viewep-flffeets j»b seeker *and programs become reality. (pp. /-»J 



Many problems are introduced when the. perceived function of a public service 
agency, changes. The employees must be prepared to ^operate differently along 
different guidelines, and theS^ublic must. become aware of new eligibility 
recpairements and new services offered, 'as an agency goesfrpm working toward 
solving sociaf . problems to, placing people in employment. The criteria that . 
are used to determine the -Effectiveness of the agency _and its ejnployees also 
change. ' -Developing an effective p3:ogram of -organizational effectiveness 
then, depends on being able to build in enough flexibility to meet these 

- changes! " \ . . ' 

Regardless of the criteria used to determine the,.effectiveness of a public 
service agency, regardless, of th;e particular goals and objectives of the 
agency at any giVen point in time, perhaips the one common- element that re.- 
mains is the empjoyee who has to implement tHe wishes of the supervisors, 
the legislatprs, the ^society . ' Maicing effective use of the human resources 
is a crucial consi?leratiq'n for the survival of an organization and the 
achievement of organizational objectives. • . 

This Report is concerned^ with the development of the Management Audit Survey 

- (MAS)Va system that gives all levels o^ management an analysis and review, 
of management procedures and certain organizational climate characteristics 
to stimulate higher levels of performance thrpugh better use of human' re- 

, sources. The MAS system is'b&sed on the assumption that more effective 

levels of human resource management wilP result in higher levels of organi- - 
■zational productivity, regardless of the criteria that are^being used at any 
^iven time to determine the effectiveness of. the organization. - • ^^ 

The purpose of ^fie research reported was to -determine how well the MAS could 
predict .various Employment Service performance criteria, .if the MAS could 
predict these criteria Efficiently -and effectively, »then opportunities for 
improvement^identified by t^he MAS with consequent action plans ajad other 
follow-up procedures to impr&ve huraah resource management woiild be likely 
to iinprbve performance by <;tfeat^organi2ation. " , 



Overview' of ^htj" Management j^dit Survey (MAS) . 
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The MAS system begins with a*;^0-it%i .questionnaire which is administered to 
a^ employees of an organization to measure 19. score areas of huitan resource 
management. These data are computer-processed., with random responses and 
other sources \jf error. eliminated, sd that each supervisor of a work group, 
having five or more employees -who completed the MAS survey, receives a com- 
puter feedback report together^ with a handbook to help interpret the scores 
arid' to provide suggestions' for iriiprovi-ng perforrfiance.^ ^ ' 

The system was developed to comp-lement exi'Siting auditing procfedures in .th^ 
Department of Labor by, pfovi'ding an economical review of management proce- 
dures thr5ughout a large organrzat ion by using descri^ions provided.by. - " - 
.emplov^ who are in an excellent position to T?bserveli^nagement practices,. 
The pil^ose of such a syst^em was: ' - ' 

• To aid supervisors and, managers injmaking beJteWuse of^he 
Department's resources to promote greater. w^rkeiS satisfaction; 



• To 'guicLe all levels of ijianagement in. identifying and over- ^ 
coining problem areas; / . . - 

• V •^« To stimulate improvement in the management areas. inost in • 
♦ "..need of attention; and 

• To enable the Department of Labor to do its work more effec- 
tively. , " . . ' . 

The MAS was not designed to replace existing auditing procedures or fiinc- 
tions^ but to provide SSmplementary informatioli that could be combined with 
that obtained through regular auditing procedures to provide information 
and feedback to supervisors, so that the individual work groups could oper- 
ate more efficiently. If the .individual work groups could function better, 
then the total system could fulfill its mission more thoroughly and more 
effectively. Table 1 shows how tl^e MAS complements th^^operat ions * auditing. ' 
The information that is obtained provides^ an extensive' picture of the func- 
tioning of the organization and the organizational climat^;" as the table 
indicates. . The MAS does not choose the best solution for problems in manage- 
ment or directly evaluate personnel. The MAS does not take into account 
those situational variabiles that influence the way a supervisor can func!tion 
•and, thu^, ali other informatioi/should be taken into, accountyin- interpreting 
the results. _ ' * " ' ' ^ 

The-l^S can be viewed from a number 'of different pe^spefctives^^ ea^h' of/which^ 
'contributed implicitly or expliciivly^t-o the' development and appi'ioation of 
the system: * . " . ' ^ ^[ 

1) 7he rapidly developing fields of operations auditing (Lindberg arid Cohn, 
1972) and internal 'auditing (Institute of Internal Auditors, i969JJ^ave some 
goals similar to those of • the MAS, syice they are also concerfted. ^th the 
effectiveness of management. . These procedures and the MAS^assess somewhat 
similar kinds of iS|(t£ormation, but through di-fferent kinds of procedures . 
Operations auditing 'and internal auditing rely hearvily on int-^Vj^ews and ^ 
statements- Df operating procedures, oFy^tives, and results obtained, which 
are often unique ,to each -organizational component. In contra^, the MAS 
uses-^a standard qiiestionhaire to obtain from an employees a description of. 
existing c6nditiatis throu^hout_„the entire organization witti ^separate feed- 
bacTc reports for each worl^ group or brgariizational component. . The MAS ajid - 
traditional management auditing procedures can complement each (other, s^ii^e 
the MAS can be applied^^to a wide range of, orgam^zational .components on a^ 



cost-efficient basis, with the results used to ^jjqitify areas in need of 
follow-up attention with other auditing procedure|.f Each approach also prcJ- 
Xrides complementary irrlFormation that can contribute to higher levels of - 



\a|rganizational perf^^anjte 

2) The general rationale of systems theory as illustrat^#5i.y Katz^and fta 
(1966>>and Baker (1973) al^o contributed ro the development of the MAS. In 
systems theory, the continued survival of an organization can only be. main- 
tained through a 4ihorough understanding pf its functions.--economiQ and i 
behavioral--as they relate^ to its subsystems of man ag-ement^. production, 
adaptation, suppdrt, and maintenance.. Since "social structures are essen- 
tia\4^ contrived -systems. . .anchored in the attitudes, perceptions^ beliefs, 
motivations, habits, and-=expectatlons of human beings (Katz and Kahn^ 1966, 




" Table 1 • ■ . 

Complementary Characteristics of the Management 
Audit Survey fey^em and Operations Auditing ; 



Characteristics 



MAS' 



- Operations 
^Auditi-ng* " 



1. Coupon Standards 

2. Organizati6n Wi'de 

3. Feedback Specific to Each 
Unit I 

4. Comparative Feedback Across 
Time • 

5. OrganizatioTiaf Development 

6. Quantified 'Indications of ' 
Unit Performance ^ 



7. Emphasizes Procedures, and 
Organizational Characteristics 

8. Provides- Suggestions for Unit 
Iihprovement ^ 

9. Chooses Best Solution 
10. Djyrectly Evaluates Personnel 

Employee In^ut 

ft 

12. Considers Other Sources of 
^Information ^ 

13. Indicates Opportunities for 
Improvement 

14. Indicates Problems for • 
* Further. Study. 



Yes 
Yes 

Yes 

Yes 
Yes 

Yes 

Yes 

Yes 
Np 
No 

Yes 

No 

' Yes 



No 

- No 

< • • - -r • 

Generally Not 



Indir^tly 
Judgmental 



Yes 



\ ? 
No 

Yes 
Yes 
Yes 

Yes" 

^ i- 



*Taken frofti: Lindberg, R-^A, , and Cohn, T. Operations auditing . New York 
American Management Association, 197Z. * 
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.p. 33) /Vdcsdriptions of variuus fuiictious by .people in the organization 
should be examined to further tjie developments of the organization. Obvi- ^ 
ously, since employees a:^e responsible for the completion of the work pro- 
(;:ei5scs and output of an organization and arc, thereby, jj^n a uniq^'nbsition 
to describe the organizational pTocedures that impact on human resource ^ 
management and organizational productivity, the use of • emplrofrees a3 an in- . 
fo(rmation source is a major asset of the MAS. Ultimately such information^ 
from^'the MAS needs to be integrated with other' information on pro(;ess and 
output measures to 'provide an integrated systems approach to monitor and 
stimulate higher, levels of performance on^the cohiplementary goals'of 
society, the -organization, and the employees^. 

3) The third and most ^direct antecedent to the development of the MAS was 
the traditional area oP^ research through attitude surveys, which has been 
replaced by a broader focus on orgafni zational psychology. .In contrast, with 
traditional attitude -surveys--which Qxamine individual needs and reactions 
to the organization--and some organizational climate li^terature--which con- 
tTiins many ^issessmepts of tangential or underlying construct features* such 
as* support oi; waVnith, or' mot i vat ional conditions , et|i: the' MA'S tries to 
attackv tunct'^ionb; and i^rocedures in human resource management directly on a 
more b&haVioral level, thus contributing to more effective diagnosis as 
well aJ providi4ig ^suggestions for improved levels of {performance. \ ( 

r4 ) ['he 'development: o'f ilie MAS can also be viewed Sfom an organizational 
dcveloplnent standpoint. Numerous programs', many at substantial expense per 
participant, have been developed to provide trailing and exp^eriehces which 
would r,esult in improved level-s of indiyidual^nd , thus^ organizational per-.. 

"formance. Typically, organizational developnjent programs have used tangenj£| 
tial or underlying approaches Which are belidved to be crucial for effectijfe» 
organizational functioning. The-^MAS systeipC which provides feedback ©n ^a^feS 

-organikational woi-k»,gf oup , together with-the Supervisory Handbook, wliich 
accompanies the computer^ feedback, and provide|g,behaVioral guidelines, are , 
aimed directly _work procedures and opportunities for .improvement and 
require, no adaptation or transfer from the developmental sequence or situa- 

'tioii to the job situation. In view*. of the diagnostic sensitivity of the 
mS and the high situational relevance, integrating MAS -and organizational 
development strategies can be anexcitipg and mutually reinforcing endeavor^ 
to bring about higher levels of development and organizatiopal performance. 



Research Objectives 



Although a great deal of technical sophistication has gone into the develop- 
ment of the MAS, the empirical evaluation of the system against organizational 
'performance criteria has been limited primarily to the State of Pennsylvania. 
Thus, the"general- goal Qf the research reported here was concerned -with exten- 
ding' and 'generali zing the resultts obtained. These results indicated that 
local offices differ in how they use the'ir TTumai} resources^, as measured by' 
the MAS and these differences were related tp a variety o*f Employment Ser- 
vice perfQrm*nce 4l?d process measures. These results need to be extended to 
other geographical areas with a: broader iijitlusioh of situational or control 
factors so that the advantages and limitations of the system can be defined 
•to permit effective use of MAS-^sults to identify and capitalize upon oppor= ■ 



\ 



timities k>r improvement. While the general goal- i^ concerned with further 
validatioiivof the MAS, a number of interrelated objectives ^re involved: 



to d'efine^ quantify, ^d describes^ selefWted performance measures to ascer- 
tain meaningful patterns oi relationships and deficiencies among the various 



I) 



"^GJiit^ria. 




/ 



^-2->--Hw~^xamine the relationships amo\ig various situational factors and the 
criterion measures. This step is nedessary to develop p^erfo^mance indicators, 
or indices, with which- local x)ffices^may be compared. ObH^usJ^y, a locdl - 
office in a highly favorable situation may look relatively a\jtstanding on 
outcome measures, yet be relatively inefficient on management^procedur6s . ' 
. Thus, the impact of various control variables needs to be assessed ' to mi^ke 
the local offices comparable on the basis of output. Thi^ would facilitate 
an analysis Of the effectiveness of management procedures' by the, MA5 variables. 

3) To deternjine the validity of NIAS measures' in predicting a variety of per-^ 
forlhance *<yriteria and to examine the relationships of MAS scores to available 
control Scores. . ' , ^ 



Further demonstrations and generalizations concerning the vajidity of the 
MAS would lead to more-'ef fective use of the instrument and managemerit of? 
human resources. In addition, previous results have indicated .that certain 
management procedures can be emphasized to help overcome the effects on 
such measures as office size and' workload. 

The reppxt will proceed by examining the research findings on organizational 
climate and organizational effectiveness w}^.ch Will be presented in Chapter 
TI. A complete exposition of the development and previous research of the 
NiAS will be found in Chapter III. r Chapter IV summarizes the procedures - 
foUowed. in this study/ Chapter V presents the results of the data analysis, 
and Chapter VI presents the conclusions, recommendations for futur^ research, 
unci the summary of the project. 
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CHAPTER II 



' ' f/" , ^ REVIEW OF THE LITERATURE • 

^ .brganizgrtianal climate, .psycho IpgicaLenvixonm resource managfe- ' ; 

ntenty ^nd organi/aiib^ai development are^ aiji phrased which ere popular in ' ^. 
^e p'rganiz£itioijaL' and- management literature The purpose- of this section ^ ' • , ^ 

" of the: report is toV siinuriarize briefly .the state ,otf the art in- these areas ^ - . 

ar\H^y are relat.ed to the Management Sudit JSurvey^ fOcusj.ng .most directly" ' / 

upon o^^^jg^niza^lflfopal climate, and to discuss the InterfaCer of the*MAS with ^ 
iiuman rteource management and the process of planned. organizational change. 
Also included in this review is a staranary of some of the research . that .has , • . 

' bedn done bri^he Enrojloyment Service and previous, IBRIC research on orgariiza- \ , 

* tional climate.-^ « " • . ^ ' • . '* \ 

, ' , * Organizationatl Climate ' 

* Four mkjor\ literature .reviews of organizational climate exist to date 
(Forehand and Gilm^, 1964; Campbell, Dunnette, LaWler, and Weick, 197p; 

James; and Jones, 1974; and HelJfciegel ^and Slocum, 1974). Forehand and . v 

Gilmer,* and CampbelK et al . , defined organizational climate somewTiat 
differently and ccflfse^iuently focused on^sl'ightly divergent organizational 
"bharact^risticsY James and Jbnes reviewef the twa<f?vt^ws, as well as stufdies 
by Schneider (1973) and Guion {1973), and di^velopecl a. ^thesis of organiza- 
tional climate research. Hellriegel and Slocum took.a \|ystems perspective 
and analyzed several climate studies relative' to it. Wltq^the focus of 
this\review will be largely *on the conceptualization' of clMi^te x)f jJames 
and Jones, other implications and research will also be rev^^wed: ^ 

James and Jones cateM^?;^^ thfe work of forehand and Gilmer as repr^sentar 
tive of the ''multiplX ^leal^ment-organizat attribute»^ approach, which « 

defined organizational/ cliirate as: " ' 

.... a, set of characteristics that describe »an organization 
and that (a) distinguish, the organization from other organiza- J 
tions> (b) srre^^elatively enduripg over time, and (c) influence 
the behavior of^people in the organization. (Forehand-and Gilmer, 
1964, p. 362) ' 

Forehand and Gilmer suggested that before organizational climate can be 

used as a <?&nstruct, it must exhibit: . ^ . ^ . 

N. . ■ . 

'l) identification of organizational units (department, divisions, etc.) 
1^0 serve as the subjects in the model,^ thus establishing the comparability 
of units across organizations;^ ^ . " I - ^ « . 

2) homogeneity within organizational units, ^ch that the objective deter- 
minants of organizational climate are applicable to all subunits; v 

13 - 



3) existence, i5q)ortai>ce, and stabiUty of* organ izationa*! properties'*, and - 

4) meaningful combination^ • or configurations of organisational prope:|5ties , ' 
such that the climate ^of^a particular orgenizatjon c^an be best desorib||^. 

Further, . ^they -stated That' the reSeayCh (existing at"ithat ^ime (1964)^ lacked 
measurement rigor, sincei there^ were" few operational (behaviaY^lly-bd^ed) 
'definitions of climate •dlmensions,'^lmost no > investigations ' Qf^validity^, 
and .the research'' designs were insufficient to account -^r individual/organ- 
izational interactions • \ s r - * . 

• '* •■' ■■■ ' • ^ . * 

Campbell,* et al.. (]f970) defined organizational climate a;5: . ^ . 

V . . a 'set of 'attributes 'specific to a^ particular oi^axiization 
that iuay be induced from the^ay the organization deals with its 
members and -its environment. F6r the individual member within 
an 'organizarion, cli'mat;^^ takes the form of a set of. attitudes' 
and expectancies jvhich describe the organization in terms of 
•both static characteristics (such as degree of autonomy) and 
behavior-outcome ^nd outcome-outcome contingencies., (p. i90y 



4 



Campbell, et al., through a synthesis of. past studies, named four constructs 
which were more or less common to Jthe^studies they reviewed. These were: 
1) individual autonomy; \} the degree of struct'ui^e imposed upon the position; 
3) reward orientation; andS^ consideration, warmth, and support.. While' 
Campbell, et al . felt t-^alfc^-rois list was incomplete, the dimensional complexi 
of organizational climate as \nown today is -far beyond* their original concept 
ualization. However, their concerns for the, **perceptual measurements-organiza 
tional attribute*' (James and lones, 1974) approach to conceptualizing organi- 
zational climate ate still critical. Essentially, these are: _ - 



1) Generalization from the perceived situation to the aqtual situation-- 
this involves construct validity considerations fi.e., tb' what extent does 
the score area-really measure the actual situation)-, as well as reliability 
considerations, such as what level of. consensus exists among employees with 
respect to the management procedure in 'liuestion; \ " 

,2) Objectively vs* perceptually 'defined factors or scores--including such 
considerations as breadth of measurement, appropriate scoring procedures, 
test construction metljpdology, arid 'item selection procedures; 

3) Level of analysis/"- involving what statistical procedures should bB- used 
tp reflect accurately the data at given levels within the organization, e.g., 
individual level data should be used to ahswer some questions, and group data 
or averaged individual level data, should be used to ai^^wer others. Further, 
this concern involves problems in the^accurate and approjpriate reporting of 
results and/or relationships across 0|H|anizational foci; and 

The criterion problem--how can the validity of the scores , perceptions, 
~ bo evaluated, what should they be evaluated against, and how accurate 
and appropriate are the criteria used to evaluate the measures. 

Schneider (1973) re^conceptualized organizational climate into a ^'perceptual 

1-^ ^ . ■: 



*tn^rlsurei?.ent-indiK^idual attribute" apnli^ch. Schneidpr defined climate, .as 
xojrows: - ^ ^ ' • . ' - ^ ^ r ' 

The coi*ept climate in tKfe present research must b e 'y'^ 
described as person^listic; climate is atr^ndividual V^^"^ , . 
*^ ceptiont There was*'ri6 a^t^empt to restrict the climate, 
definition to ^pe'rceptions shared' by members of - a worx-y; , 
J group or organization. As st-Sted elsewhere^, . .-"what 
is .psychologically important 'to the individual, must 



'how ]\e pjpJrceives j^is work envirc5mnent,,^ot how oth^^ 
, might ^choose to describe . i^" . (p. 254) 'r /// \ ^r - 

Schneider further ' stated that' th^ data^'collected ^hould be' appropriate for- - 
the^level of explanation an^^ that shared -perceptiofis of climate m'aybe im-j^- 
portant for^i7edict;lng the' behavior of many individuals. . 

Schneicjer^s' major con^rns^were the match betweeq/cbncep^uali-^dtion- of c-lij^^ . 
mate and' what, measuxememt methodology the concept implies, ^nd^the relation- 
ship among/ejiqployers, employees; and customers'/ The first concern implies 
level of analys^^, and the second ijnplies system differences, i.e., the 
organization's system orientation (open vs. closed) and its effect on the 
people who enter its environment. y ^ 

Guion (1973) considered" the family of constructs implied by organizational 
climate to "be one of the most impo'rtant to enter the thinking pf industrial- 
organizational psychologists ''in many*^ years*' Hp- 120). Concerned about organi 
zatiojial climate research-, he -further /stated that: ^ 

, . ■ . J' / . ^ . ■ 

^ ^The.^ea of ''organizational climate" appears to refer to- an 

Itjp^^^';^. or set of -^tributes , of the work environment.^ The ^ 
xdea of a ^'p^Tce?ived organizational climate" seems ^ ambiguous ; 
^e can not be sure. whether' it implies an attribute of the 
organization or of the perceiving ir^dividual. If it refers to 
the organization, then measures of perceived organisational ^ 
ciimate should be eva^rua^d in tefnfs of the accuracy of the 
^ perceptions. If it refersNto the individual, t'hen perceived 

organizVitional climate may simply be a different name for job ^ . 
— satisfaction or employee attitudes'. (p. 120) V 

Hellriegel and Slocum (19743^'ad4ed no new criticisms, but they emphasized, 
the^systems approach and^he confusion about satisfa<;;tion and climate.* They 
defined organizational climate as*^^*^ -h. > - . . 

. . . a set of attributes which can be perceived about a 
particular organization and/or its subsystems, and that may ^ 
• be induced from the way that organization and/or its subsystems 
Seal with their members and environment. (p. 256) ^ . ^ 

^> ' ^ • ' • ' . ^. 

With this set;, they further structured climate to fit primarily into the 
"per.Q^^ptual measurement-organizational attribute" approach. 

To. summarize, then, three conceptualizations of organizational climate have 
bejen represented. These are: 



1) the multiple meaSuremetit-organizational attribute approach (a^ exemplified 
by Foreharfd and Gilmer, 1964); - ; 



2^ . the perceptual measurement -organ i2:'atiQnal attribute approach C^s exw- 

plified by Calnjrbell,, et al;, 1970; and -by H^llriegel and Slocum, 1974); and 

'"•^ ■■ / I . ' 

3) the perceptual •^easurememt- individual ,atti-ibute approach (as outlined by ■ 
Schneider, 19.73).. J ■ \l . ^ ^ 

While ,the^e c|a(nce'ptuaLi^tions are not independent-, they, have generally 
}ed to different research methodologies, different^ levels'-of measurement, 
-^nd different s^t's of variables, ^owever, many similarities do exist among 
the coaceptualizations as fcan be seen by ^he overlapping nature of the authors 
research concerns. ^ Acrossi climate conceptions, "these conc^ms all fall mo^e-^ 
or less into the following categories;' v ' " A v * 



/I) J^ork gjjipup differences and identification; j- 



2) 'shared' concensus of perceptions; 
3) 



relative permanence', and stability of the dimensions studijed; ^ ^ 



4) lev^^of analysis l^^qregation of dal^) ; 

5) perc^tual validity and' 5|^eralizati\^n; 



6) organizational systems perspective (level of organization); and 

7) attitude-perception overlap, or satisfaction-cli^^^xonfusioM. ^ 

^The alarny-Tig thing about these concerns |s that,*»in 6v^r a decade of research , 
on organizational climate, few of the measurement and methodological problems 
have been resolved or even received sufficient scrutiny to result in greater 
understanding. Some possible explanations -for this lack of progress are: ^ 
1) limited research group intra-communication; 2) insufficient knowledge of 
past research; 3) a lack of sophisticated measurement too^s and methods; 
and 4) poor understanding of ^climate -complexity . Further, very few studies 
have .actually reported significant validation results, even though this is ' 
critical if the research methods and the results of the studies are to be 
shown to have long-range utility. Agreement within work groups and between- 
wofV group difference's, have also been mentioned as being essential tcf^an 
understanding of organizational cimate, but no data--other than that on the 

^MAS--have been reported. . . - 

Previous^esearch has resulted j.n. some confusion because of the lack of 
clarity in procedures' and goals,^ with the findings of the research not Seeing 
convergent. * -For example, the content and focus"^ of the measuring instruments.^ 
have not been consistently and systematically organized, including diverse 
topics as individual reactions to tjie work environment-, descriptions of the 
work group,- , and general or abstract characterizations -.of . the. total organizar _ 
tion. Many of these may'*be appropriate if treated in a systematic fashiony 
so that clear implications c^in be obtained. The MAS scores focus most direct!) 
upon management practices (Delegation of Authority, Planning and Administra- 
tive Efficiencj^, Training Effectiveness, etc.)» with* some individual level 



V 




scores (Satisfaction with 'Pay, Work Satisfaction , etc 0 included to -define 
dif'ferenses between these different typ^s of scores. . ' 

These issues, in turn, are related .to the leyq'l of analysis to which lihe ^ 
* data' are aggregated. Data dealin'g with individual reactions and preferences 
(att.itpdes'j are obviofely most meaningful when, the individual re^pond^nt 
is the unit of analysis, while the interrelationships of work group-focused 
climate measu/es and criteria »are most meani«gf^ when" work group means are 
used^as the unit of analysis. Data on individuals can be aggregated to^ 
highe-r organizational levels' 'Avith meaningful results only if there are orr 
ganizdtional sr^ffetts wKicTi result in. significant differences among groups 
at that Higher level. Data oh individuals aggregated to the group or ^ * 

higher level will approximate a r^dom' distribution of means with limited 
meaning, unless there are significatnt effects on the group means„ such as 
management practices, selection^rocedurea^, ©tc. -la^ addition, the nature v - 
of what is jneasured may chang€^s ttffe data are aggregated to higher levels. . . 

The Department, of LaJxJ^studifes using the MAS have begun tcx yield an under- 
standing o^ tiie re^^rch problems involved and are presently unco\^ring. the - • • V* 
dimensions nece^ary to provide measurement technology d'eveJ.opmdht with • 
3'ufficiently omplex data over time. While the MAS questionnaire falls . 
•primarily inro the ^'perceptual' measurement-organizational attribute''^, 
approach. to organizational climate conc'egt;ualizat ion, the MAjl has cu\ 
across the other two approaches and integrated thqm in different stuc 
and analyses. These studies and -analyses have also cut across the research 
problems stated earlier, investigating organizational climate and differences 
in inanagement practices across work groups, with a variety of performance 
criteria, . , ' , * r ■ 

To illustrate further, in the MAS research the intraclass correlation/qpef- 
ficient has been used to define the degree of consensus within work groups 
and thus provide information concerning the level to which data should be 
aggregated^ Since ,wide differences between work groups have been obtaCrned 
pn MAS scores, 'higher levels of. data aggregation, beyond the work group, 
mask group differences as high and low groups are combined. Typically; 
high levelS^of consensus within work grqups- and meaningful differences among 
work groups* are obtained vyhen the item content is focused upon highly ob- 
servable organizational procedures or characteristics. For example, the 
MAS score" of Physical Working Conditions has the highest level of agreement 
among work group members. The results obtained with the MAS indicate that 
the^ system is Jiighly sensitive to differences in management practices across 
work groups which are fe;Uably assessed -according to the agreement among 
observers,. , • ' ^ 

To summarize, these approaches and procedures used in research with the MAS 
make possible- some clear distinctions between organizational climate research 
and the* area of traditional attitude measurement. Specifically distinctions 
between climate research and attitude measurements can be differentiated 
according to item content, focus* of measurement, degree of consensus, l^y^T 
Of. anaiysis>-and val idi tyi^.against_ prgani.:iAt^^^^ perf omf nee criteria . jhe _^ 
MAS scores have shown significant relationships to a variety of performance 
criteria, where attitude surveys have been very deficient. Therefor^,, the 
Mils studies have led to a reconceptualization of organizational climate in 
line with the ^''perceptual measurement-organizational attribute*^ approach, but 
extending it, ' . 



TTie ^lAS is kljo highli compatij>^:«s^ith the needs of e^loyees and manage-' ' 
ment— by producing a fWdback systVr^hich' pfravijdes the necessar)r links to 
tie together an upward \cHDinmuni cat ion system for Employees with organi-^ational . 
interventions and individual Jb/^vior regulating systems, such as management 
by objectives (MBO); oir organizational [development COD)i and the econoi^^ic. 
regulating and reporting systems- -the criterion measures af organizational 
performance . ^ 

Vhe integration 9^ thesej^a^st^ns within the organi?atj.on r^Jl^ires a systems 

' thfe 



e integration 9^ thesej^a^st^ns within the organisation re:j^^res a 
cqnceptuelization of/tkTorgani^ation According to plover (XgyS)-, 
rationale for thiV^proach is t^at : • . \ > 

■.iu^-- . ■ ■ ' • - - / 

The .conoiJ^wi^^^* ^jr^^v^m^ ..^^^ — >y ^ ^ 

ceht^]^mp0^t:ance in the thinking of social scientists and, 
^ morey^cificsilly, organi'sational theoris^ts. <. Systems thinking , 
■ y^ught to be .^more meaningful w^y to look at complex 
fenomena, differs ifrom the traditional analytic method o£| 
"Sroanalys-Jtt:^/ studying/the processes linking the paxts ' r 
together, -shift .in emphasis from analysis of Rarts and . . - 

quantification of casual relations to knowledge ofjfhe wholei 
not by observing parts, but by observing the procesfses taking ' 
place within the whole has r'tesulted^in a different view of r v. 
organizations- Jhe systems approadh maintains that the best 
way to view' organizations is to vi^ them as systems';; with 
y emphasis on the inner relationship akd interdependeney of : . 
Mparts, (p. 1) ' ' ' ; /' /i' • ^ 

Thus, ^an integrated system pf measxi^ent models and methods or informational 
subsystems (each with demonstrated ^uniqut^ and/or copnon- ?^iterion validity) ; 
would lead to more effeqtive oi?ganizational functionitTg ah<^oal attainment ' 
through increased capability ^^4Ii^ytegrate all. of the fjipctioh^l subsystems. 
Katz Kahn (1966) have described tliese subsystems as production, support, 
maintQ>$ice, adaptive,- and managerial-. Each of these siab?y'stems has a function 
to perform for the larger organizational system, Some^^^mbinatio.h of the 
systemic models and methods of MAS', MBQ, arid OD would tend to maximize^returns . 

Before the scope of organizatic^nal climate and its intracacie-s^^caiTbe fully . 
grasped; understood, and develc^ped^ the followin/;ire needed: 1) continued 
development in measurement tecJinoXogy; 2) studid^f sufficient com^ejcity 
and duration; and 3) models anc'fliethods for implemS^^ng organizational cli- 
mate' findings in -planned organizational change. Such achievements must 
await the findings of future r^seatch, some of which is outlined later in 
this report. 



Organizational DevglopmenJ: 



scjitfiard (1969) defined organizaltioiial development an effort " (1) Planned, 
6rganizati6n-wide, (3) - managed from the top,, to .,(4) increase organiza_- _ 
tiotial effectiveness, and help through (5) planned interventions in the ' 
6Fg*izatioif processes ^^u^^^ 

organizational development is to change the system. Again, the emphasis 
is placed on a systems perspective. Schein (1969) amplified this pomt and 
illustrated the ,ne^d for the management of human resources. in a systems sense 



. I am'^not contending that f bousing on human processes is the only 
path to increase organizational effectiveness . . ' r"h^ arguing/ 
* however, that th$ various functions^ whicW made up an organisation 
are ^always mediated by the interactions, cif people, so that*-org^ni^ 
zations cannot escape its human ^processes occurring between them. 
' ' Therefore, it is oWious that the better diagnosed these processes 

are, the greater will-^be the .chance^ for finding, solutions ta 
\ technical problems which will be accepted and used by the me/nbers 
A)f the ^organization. (p. S.) , ^■ 

p/enlh and/sell (1973) extended th^ definition of ,oVganizationakdevelopme7)t 
such that it included management of organizatibrta^culture. Hus^(1975) .A 
included climate in this defini^on and staged th'^ systems which don't V 
make fuil use of modem social knowledge dnd ^echno^.fij|gy' a^out. human motiv^- | 
tion cannot appropriately •^change the climate. and Bell also stated' 

''that there are three basic steps which musrt'pr^eede the institutionaliza- 
tion of these new social techniques^ - entry, TOxmative change, ^Scn^ structural ' 
change"' (p. 13). ^ ^ / , ^ • ' , ^ 

French and Bell (1973) and powers t]Ci§73) discfussed the survey- research- j 
feedback system. Tli^ey primely focused on this type of '^aclKipn reseaorch," 
and data feedback in workshop^ ^ Cycles like this are general IjKwhat is 
needed to process the inputs, ani^out£i^< from the'.integrated measurement 
system in d^.scussion'. Surveys frpm each' measurement/tnethodological realm 
yield the data for- sy^ematic diagnosis, research, and evaluation^ which should 
lead ideally to^ a feedback system which feeds back to the appropriate user- 
manager econometric, behavioral, 'and economic-behavioral j.nteractive data * 
ahd^ction alternatives. " \ ' 1^ 

In an OD Evaluation design, the MAS would be administer^ just prior to .the ^; 
beginning of any iAervention into the organization. The resulting profile 
across the 19 . s^cores for each work group can be used di^gnostically to 
determine what areas need to \)e worked on in each individual work giu^i^gi, 
or any other OD strategy may be ysed. Since these,, organizational climate 
scores. can be successfully related to and can predict hard organizational ^ 
p^formance criteria,, an effective and successful OD intervention strategy 
may result in increased scores across the work group profiles over time. ♦ 

' \ . ' ' t J 

The MAS system allows interventions on individual work groups or an organi- . 
zation-wide lev^ just after tKe managers receive the feedb^k (with or 
without the use of the Handbook for Supervisors) which alloWj^. some limited 
types of experimental designs^ using control groups (e.g.*, intervention on 
50% of groups, no intervention on the other 50%). If the interyQntioi;i' or 
treatment is made on an organization-wide basis, then the organisational ^ 
norm base for producing the two types of MAS scores should increase ov^er 
{Itime to. indicate successful treatn\ent (pre-' and post-intervention ^^plica- 
tion of the MAS). V - • ^ . ' 

If the intervention is made, on only some 4^^ork groups, then work groups.. in 
which intervention is made should obtairWilgher post-intervention prj3files 
than work groups which recejpred.no^^^^t^ 

this design. Both treated and untreated work Iroups may increase over 
time for reasons unrelated to the intervention]! so this design allows the 



consultant to attribute the differehtia^l increase in. the profiles of* the 
treated work grdups to the treatment rather *han to unknown for.ces, 

^ . * •■ 

Since the MAS provides 'feedback ^at ^eath J.ev.er\)f the' organization, the sys- 
tem can be easily implemented in organizational dfeyeji)pment intervention^ 
strategies: .Glover (1975) stated that "the intffe^rating mechani^' in systems 
is the interrelationship ietw^en the pafts of '^th^e systeifil Th^ hiW^rchy 
within the system due to the ascending order of compTexity result^ in the '^'"^ 
i^ntermediate structure at each level which coiy^jtrains; arid cojitrols all its^a, 
-Subordinating systems. This. characteristic of systems resulxs in the goals 
of each subsystem being related to the larger set of gOa^s of the system" ' 
(p. 7). The problem^of goal integration, managementv^ContrSl, and. ef:^ctive^ 
intersystem communication ^of the ES --are/ treated in tiae next section^ ^s^_X 

' ' y ■ - ' ■■ ■ 

' Studies the Produjbtivity of the ES . 

Prior research studies of the pto4uctivity of the Employment Service have ^ ; 
.usually . focused qn Subsets of variables that may have ail.'feffect on the out- ^ 
ames of the E& process , ^Sorae studies have concentrated on^ ejj^^mal. factbrs 
that influence ES operations, othe^cs have .concentrated on ^M^^al^^^ .; 
Others have been concerned with the type , of client" tha'J; ojot,^^^ ES ^or 

services., Studies a$sefising management V'^^^'^^^^^^Mlf^^]^^ 

ES outcome performance criteri^^tSve been limited^'^^S^^^^ -J ' • 

jMuch of the research dealing, with the productivityl;.pf;> 
' around the Balanced I^jhacement Formulia (BPF) ; The i%||^^^es 

. effdrt to. allocate- Federal funds oit th§ basis of ^defliorkt^ 
Thus, the formula serves g^s an incerit'ive for impfovaJnent in thef^cjija;^^ 
quantity of s eryici^i^V, rendered . Equally, important, ;t|i^-^^ 
national policy anii 'lEiili^reb^ serves t0 "define th6^^^ of the org^in^izlkt^^ 
and to promote the integration ot these^- goals tJfroU^ the^ot'ganizatvon 
.In this process, the BPF provides^ a basis for )t\e state agency^ to evaluate 
its r6^1ts. Considering the coraplexiiy and di faculties involved in 
establishing effective procedures fro]i the national level through state 
agencies and intd app'rfeximately 2400 Tpcal officesL,.. these goal definiliioris, 
integrations, and reward and monitoring functions 'are extremely important. 



The fopio^ajwas applied in 1975, arid sulteequently^modified in 1976 to reflect 
a greater emphasis on quantitajiye and; qualitative functions in placements^ 
However, the largest .single con^oljent in the formula during both years h|as 
been the number of placements per mn yeai*, which has been extensively studied 
to understand its dete^inants and /causalities ;^ For example, Englanderj (1975) 
found that there was a significaivt: relationship ^between individuals placed 
per man year an^ th6 perc^nt^.of^"^ employment in 'manufacturing--corroborating • . 
the finding of Fong (19750- ' ahese two authors concluded that' the ES ip 
better suited to provide placement assistance in those 'areas where the (economy 
is more oriented to. ipanuf acturing activities. \ . 

Englander al^ fqu^^ 

placed per man year and the workload of the staff In. an office. Apparently, 
the larger the number of -applicants and /renewals with whom an ES staffer, 
works, everything else being equal, the easier it is to have higher produc- 

•■ ' -14. • :• ' . , ■ 



tivity on this critex>ion. The larger^ the worMoad, the gwi'ater isJ the like- 
lihood of h;atr[^8 applicai^ts whose qualitiesj^nfl experience make them highly ; 

"^employables and easy to.place^ However, in this sitfuation, the percentage 
of a^l appHcantVplaced is lower. Thus, one goajyls attained at the expense ^ 
of another, ^whiclAemphasizfes the fact tl>at. multiple criteria should be used 

' iji investigating the effectivatw^^oT an ES office, and aS the basis for the 
.goals of organization 7. , ' * -1 j 

'J-^Workload is an exampl^ of a variable that^could b% used to enlarge a state's 
- share of, natiMal Employment ^Service funding"^ the s£ate. ES(^dminlstrator. 
Although some of these variables would ,diffe¥ffi*om stdte^o state, research 
has shown that short-term placements. (En£lande!r, 1975), »w-wage plTacements 
(Center for Applied Manpower Research, 1973) ,^ concentrating on placements in 
.^ixiimfacttoing (Center for'Applipd Manpower Rese^r,ch,. 1973: Englander,. 1975; 
and Fong, 1976), youth j[uiKler 22 years of age). (EnglahderV 1975), and percent 
of minorities in the l^kr&r force (Fong, 1975) are other examples^of variables 
that>could 'be m^ipulated to obtain a higher ^anced Pla((ement/Fdrmula ' . T 
scoj^, andLmbre Federal moniets^ ' -> ' >^ . 

< ^ ■ ' . ^ ^ - . ■ . - - ■ 

>The Sh*ley Report (1975) $tjidied the Balanced Placement Formula i1\ great 
detail and -found that the BPF couX4 be .improved, and recommended that:^ 

1) individual? placed pe^ man y^^r 'be maintained "as- |i^|gxfDrinance criterion; 

\ .2) individuals, placed as atpercentage of "the ntanbgr-^of; unemployed indivi- 
duals should be used as, a performance c^iterionjand 



3) job openings filled as a percentage, of nonagiftcultiif^wage and salary : 
employment also be instituted as a performance cri^^erion-N. 

Shelley found that the BPF allocation' was not significantly affected Sy"the 
Uercerit of veterans, DOor, handicapped, UI claimahts, and older workers 
■\placed. Another key findihg of Shelley was that the performance measure^ ' 
weue infliiilnced by vafcLoiis environmentaljFactors . AboUt two-thirds of the 
variation in productiiMty across sptes was due to the environmental factors. 
If thi>6 is io, care mu^ be taken nojt to penalize the budgets of lower^ pro- 
ductivity slates'^ when the lower productivity is due to circumstances beyond 
their cbntyfel. The unemployment rate is an inadequate reflection of the 
Influence 6f these external factors. . For example, the Center foT Applied 
Manpower Research (1973) found that the unemployment rate exerted a signifi- 
cantly negative influence on a state agency's placement productivity, but 
Fong (197^1 did not find statistical signific^r^e for this variable. 

Englander\(1975) found a significant negative relationship between indivi- 
duals placted and per, capita income. Englartder .also stated that a given 
group mightVbe difficult to place either because it requires a considerable 
amount of counseling and other support services or because the attitudes of 
would-be employers towards that group are not favorable. 

.. 0£ ■ the various authors reviewed, only _Weiner and Ppwel (1976) mentioned the ' 
social value of an ES placement. The BPF does not take this into considera- 
tion, and providing a quantitative measure of. this social value would have 
tp be subjective, so -it would not be an integral part of most researchers 
investigations of ES effectiveness. If the fuijction of the ES is seen to be 

^ 21 " ' 
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that of a labor exchange', providjLng a,, match besJS^eeji" applicants and their 
appropriate labor market opportunities both in the labor force and out of 

* It, then the labor exchange outputsf should.'benefit job seekers,- employers, 
aod society. * . . , . 

The sociaii valu^ accrues not- only monetarily, but als<>ysychologically, from 
an individual's being able to contribute something wortmwhile to himself and 
to others. Not being able, to be a contribtitilg member ®f so(fiety has an 
"^imfiact on every other ar^aTot a person' s^fy, .and theri is no way tljgt this, 
can be quantified. ' " . "'■/''' 

' ' / ■ , • ' . 

Even without considering the social value of "ES -placements, Moore. (196^conA 
eluded that the dollar value of benefits received from ES placements^ptojably 
exceeded their costs. Moore looked at ^ the question of general pro^uctiyity 
on the ba-sVs of value received for resources expended by deriving^ ccJsi^- 
benefit ratios for placemei|ts by the sta€e Employment Seing-ce. He found 
that the tost per placement varied substantially from 'state to state, probably 
more. widely thali the" value of benefits 'fr6m state to 'state, but that the ES^ 
does provide a worthwhile seijvice at reasonable cost . ^^^^ 

TheN|mployment**'s.ervice continues to be an object of ^criticism, however, 
regardless -of demonstrated levels- of productivity. This was recently poSnted 
out in ai^rti^e in The Wal 1 Street Journal (Mi H er , 1976)., which noted that 
the Emp^(^ertt ServicFTs seen by emplbyers and employees ^1 ike as ah agency 
for low- Jci lied laborers. Despite Federal legislation that Federal contrac- 
tors must list positions with the ES, many companies do not do so, and look 
upon" the- ES as a pressure group, influencing €he type of employee that can 
be ''^^^ company. 

These studies of .the productivity of "the Employment Service have gone some 
distance in explaining how the ES functions, and why certain problems have 
aris-en. HoweverV the studies have not been very .inclusive, and have not 
developed strateg^s for 'improving the productivity of the ES. A somewhat 
mare inclusive study is presently being conducted by Van de Ven (1975}^ 
whose objective is to measure scientif jcaJHy and explain how situational and 
' organizational characteristics of Job Service 9ff ices and Units affect per- 
fbrmance over time. The organizational factors being measured include: 1) 
situational factors ;\ 2) overall office structure; 3) structure of each unit 
in office/bureau; 4) (linkages within and between units, levels, and other 
agencies;, and 5) performance. - - 

Any study of a public organization, as opposed to a private industrial 
organization, is influenced subtly by certain basic differences between the 
two.' These differences influence the results, the theoretical explanation 
of the findings of the study, and action strategies that can be taken to 
counter any negative findings. Giblin fi276) noted that there were 5 basic 
areas that had to be considered, when stud>?iiigaj)ublic organization: 

1) Organizational variation— the public organization^'volves a greater 
variety of individuals and groups with different and often 'mutually exclu- 

• sive sets of interests, reward- structures, and values . /ole-^onflicts- 



r 



sive sezs or inLcicsua, xcwoiu 1 . r ' 

between legislators and high-level administrators, the ^raraitment of career 
officials to pet programs, a weak chain of command, and the number of interest- 
groups are all part of the organisational variation in the public sector that 
influences x)rganizational development. • 



2) ^png-ran^ planning-.- t4»e-uiicertainties of thefyea:^/ budget process make 
long-range plrannigg a veVy tenuoiis practice. Mos|>^blie agencies basically 
do their planning on a year-by-yekr basis. Because the continuity of organi- 
zifional development is always in doubt, the influence of OD on public organi 
, /zation^ is xeduced. A *• ^ 



31^ The Civil Service s^tem--cQinpromises th'e'pubJrC adratnistratdr's responSj^X 

■ - -. — hire the ^ 

eVels. It 
id*^ limits 



* bility to manage his organization, by limiting his authority to hire the 
^ persons he wants at all but the/very highest (i-e., appointed) l^els. It 
J ^« o-iifT A ci-a-P-P fn-r nnnar nerformance. and lim 



limits his authority to discharge staff for poor performance, and ^ limit* 
his discretionar/poweKlo advance staff for outstanding performance. The 
&ivil Service s/stem, i^ effe^:t,• unionizes the entire drganization. 

4) Crisis atmosphere- -has ^resulted from the external attacks by many client 
groups that have taken legal actiion against "human resource agencies" that 
^"do not seem to serve- titem effectively. In such a brisis atmosphere, public 
\ agencies 'tend to place even greater stress oii procedural regularities and < 
aution. This excessive reliance on established practice runs counter to^,* 
ID goals, which stress management- according to relevant objectives. I 




Organizational "style" an\l effectiveness— <hSt static style, low prjipen- 
for program change, and. general inef fectiVeneWpf most public organi- 
zau5m5is render them very poor cWndidatqs for the realization of successful OD 
efforts- To -a considerable degree, this "style" is a function of patterns 
of administrative regulations, which are usually spelled out in minute detail 
in slegislation. " - - a ' 

In addition to these five areas, Aller, Mayall, Mitchell, and' Roberts (1975)' 
' . found another area that needs to be considered in any study of an organiza- 

tion such as the Employment Service: • 

6) The reliabilit>rof the data-all of the ES-generated data have to Tie used 
cautiafislys since there is some question as to its reliability.. There is a. 
stroi(g )in'ceWtive 'to undeirreport new applicants, because the time required Jtp 
fillStit ESARS data formfe significantly reduces the time available for mofe 
placement -^ikented activities. As a result, forms are freJ^uently not filled 
^ out for iiliViduals fo/ whom little chance of ptacement. is seen. The extent 

\ of this kinddf^mder^eporting apparently varies substantially from office 

to office. ' ^ . 

Despite these barriers to research concerning the Employment Service, and 
the criticisms of the system from various sources, research concerning the • 
productivity of the ES continues on ipany fronts and on many levels. Reports, 
by Frey (1976), Giblin (1976),. and Meike, Pyles, Kauffman, and Horowitz 
(1976-) have begun with definitions o£ the mission and functions of the ES,^ 
\ "^and have attempted multivariate studies of the effectiveness of the ES. tv 

realization that the objectives of the ES need to be examined, and the 
■ ■- • ■ realization that many different criteria, control score variables, and pre 
" dictors need to be examined aje all hopeful signs that the research on the 
Productivitjy of the ES in the future will produce more meaningful and more 
relevant results. 
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. IBRIG Studies of Organizatiojial Climate ' \ 

• - - u ■ ■ ' ■ • . f ' ^ 

^^^Ifi^IC's e^arliest investigation of organizational climate (Ellison, McDonald, 
^) , James, Fox, and Taylor, 196a) xnv9l^ed an- extensive analysis 'c|f various cri- ' 
terion mW^ures of' performance and' appropriate control scpres ^in a medium- 
sized government research laboratory . j The criterion jneasures inclwj^^. 
supervisotry and peer evaluations, quantity and quality of publications, 
'awards won,* salary a(fvancemen\ measures^.j^'Bt^ . ; while' the con^rbl scpres 
included such variables as age, • education, experience, size of work group, ^ 
type of Research conducted/ etc. With this foundation oi measures of x - 
/ , research products and rgg^grch .performances, ai^J^^tensive cliip^e question- 
naire was developed wMch^ lL7^t?§*^ftdM^ of style of supervision;- ' 
compatibility of indiv^idual and organizational objectives) gitoup intef-. ^ 
relatiOnsHiTps, self-descriptions, etc/ The relationships of -tbese climate'^' 
• questions* to the performane^e^ijlerion measiires were analyze^ Jp determine ^ 
fhe factors which facilitate, inhibit , or are associated with different *^ 
^ ' kinds of contributions made in. a research labora^tory. ^ 

The. results of this study indicated that data coilcq^ed "W^th the reaction of \ 
^ ^ the; individual to different aspects of organizational clifliate were more pre- / 

dictive of scientific performance than were straight descriptions of the. ^ 
characteristics of the organization, supervisors ,^eer ^relationships, etc. 
Information concerned with c|he, discrepancy between the acrtial situation, ^ 
as the scientists described lit , and the ideal situation, as the scientists* 
would ^like to see it, were as predictive of the performance measures. 

The analyses df the ciata from this study led to the conclusion that certain 
other steps had to be taken to develop axi. instrument that could measure ' ' 
organizational climate to the degree necessary^that changes in hiinan resource / 
management could le^^o improved organizational climate, and, consequent ly,"* 
higher levels of organizational perf6ri|mnce. One of these steps was level 
of analysis. ^ Instead of determining tirc organiz^tiori-wide level of cfganiza- , 
' tional climate, the instrument should be able to measure\the climate of- the 
work group ih which the' individual is located. Several work groups which 
were under the general supervision of a higher-level i supervis^jf could be 
combined to give a nlore^global indication of organizational climate. 

Subscores or scales based on the content categories of the individual items 
^ were another step that had ta be done. These ^ales would measure various 
facets of organizational climate that have an effect on the functioning of 
the 'individual and of the work group. 

The results of this study arid the recommendations for further developmejit 9^ 
tHe climatt? instriiment led to the development of the Management AuditTSurvey . 
The MAS is described in great detail in the' next section of this repott . 
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. , CHAPTER III ^ 

MANAGEMENT 'AUDIT SURVEY " 



JHE 



Introduction 



Whei\ the Managemeftt Au^if Survey wa?^ first col^^eived, tjie genera!!' goal was ^ 
to make a preliminary^ review or sio^ey^of management practices, using em-. ^ ^ 
ployee descriptions t^ identify problem ^arpa orgsmiza^^ ^ 
which could then be subject to more thorougft-audit procWdures, The survey 
of mana^eteent practices was to be applied hTO^dl)Hrto la:^ 
I compoj*ents\at an economical rate using Advanced computSrl t^echniq^ At 
AthatftiTftei^^ift^ survey questionnaires were in use) but/mey were not tech- 
i^lly sophisticated, nor were the responses to the cpieWij^nnairps effici- 
eiiJtly processed throoigh the use of computer technology. As a result Of pre- 
jious studies in organizational climate and sophisticated date processing, 
capability, the Institute for Behavioral (Research in Creativity (ipRIC) was 
contacted to explore the feasibility^^pf developing a system .that would not- 
have fhe limitations of the then available, instruments, . ' 

Although th^ analysis of 'management procedures was a topic of strong interest 
tp' both tjie researchers and the contractors (the Directorate of Audit and 
Investigataons), the combination of audit 'and organizational climate initially 
represented a;relatively unique blending of ^).erspectives'; Duting.thc course • 
of subsequent discussions, however, extensive common ground was established . 
•to make the development of such an approach bothi'feasible and attractive for 
its applications and ^^o.tential ^utility; e,g,, in improved management, using 
employees as dbs^^ers of orgaiiizational procedures ,'.improving performance, 
and, "in gf^nerar, stimulafMng more effective use of personnel resources. 

Cbhtihued discussions, pilot studies, and a preliminary validation study 
resulted iih the- dfevelppment of <he Management Audit Survey system as des- 
cribed in th*5 chapter. The system ^oper^tes by providing an analysis' of 
management procedures of small work groups at the bottom of the organization 
up to the top levels and includes subordinate components of work groups at 
each level of the organization. That is, higher level supervisors receive _ 
reports ^Ltheir immediatQ. work group as well as a consolidated report on 
all subordmate work grx)ups. In this manner the organizational climate of 
■the .work g^ups within that organization and of the organization itself can 
be examjined. The employees were selected as the grojup to survey, since^ 
they ai'fe the most directly affected by human resource management policies 
■and practices. The employees live with these policies and p;ractices, and 
observe them on a daily basis, and thus they* are in an excellent position to , 
provide an overall picture of the nature of management practices' in ian or- 
ganization. '.-^ : , _ ' : - 

The system surveys a wide Variety of management practices. The 1? score - 
areas covered by the system are generally applicable across widely differing 
kinds of organizations. The individual items are generally concernec^ith >.: 



perceptions of relevant management behaviors, as opposed to individual feel- 
ings of satisfaction or reactions to management bfhavjrors whith were retained 
in only a few score areas': ' .. ,> 

/ ■ ' ' . ' ■ ■ " . 

A scor^ for' each' w6rk group, or orgajiizational component, is obtained'ty ^ ^ . 
analyzing the answers of the employees to the questions in each management 
area. The average scores for the employees in each work g^oup;a^e:then colfi- ' 
pared to organizatioiywide and divisional averages, using the twpyscoring-^ . ; 
systems described below. Each work gr6up,af 5 or more people is'.jncluded in^^..;- 
the analysis . The N^restrict ion ensures the anonymity of the responses, of^^ 
the employees, and averages the responses, so that an extremely; deviant indi- ' 
vidual will not unduly affect the score. ^ \ . V \ 

The data collection h^ been handled on a state I^X state basi^, with the. pro- V 
ject moortors of the Directorate of Audit,. and ^Investigations of the^ Depart^ v/^^^ 
ment of La\>or providing both formal aind . informal guidance to the participS^p'g'; 
state personnel. The^ state perscmnel are in charge of the actual admini^^^ 
tration"Q,f the MAS. Each employee thWughout the state i§ giveti an adcli^e^s^^^^ 
envelope in which the completed answer she^t^ ?s plabed. ^^lis; procediire^ e^^^ 
sures the..confidentiality of the response^ of the -individual employee: 
envelopes are jiailed. to IBRIC for scoring "and analysis / Upon their a^^ 
the answer she^t^ are examined^d "cleaned up" to ensure .^Jiat all 
responses are ireiad by the optical scanner. ' ^ - W^/^i^^ 

.. ■'■ ^ ' • ■■ ■ , - ■ . : \ 

There has been a^ return, rate . of over 90% from every! stat^ j^^^^^i^vli^^l^ 
for one, which had a return rate pf -80%; , '"^ ^ . . . 

cates the general efficiency of the system 

acceptability of the questionnaire to participating emp^b^^^^^^ 
Scoring Procedures of the MAS 

Two different scoring methods are used to report the MAS.results. .Each^ 
scoring metftod provides i comparison, for each ..work group a^^^ t Wo differ- . 
ent organizational components. For instance, in the .pepartmen^ of Labor, 
each unit was compared on each score to Department ^wi,4^^^^^^^^^ and ta.>^geiir- 

cy-wide averages. The scoring methods are: < ^ 

1) Percentile ranks. This scoring method takes into accoun^^feisitive, rieu- .; 
txal, and negative responses and permits the supervisor to se^ow^ll a^. ./ 
particular unit did in comparison with all of the .units in the De^^gment or . - 
agency. In this scjoting method, the Departmenf or. agency average- i^''^^^ 

the 50th percentile^.'' If a unit has a score of 50 or better in any manage- ' 
jne"nt area,, that/unit scored as well or better thkn the average of all the ^• 
employees in the Department or agency. Any score within the range 40-60 
should generally not be considered as being either particularly high or. tow. 
since they are close to the Department avera)ge. , • 

2) Percent favorable score. This scoring method. shows the,actual percentage 

t f avorable ' respon s^ chos en -by ' emp 1 oyee s ., in^des crib ing a majiagemen t a_r ^ ... 
their organization. In this scoring method, the two most positive respon- 
for each question are used. This method provides information about. the 
rail level of- performance in each score area for the total ,organiza1:ion 
and smaller components. This is. in contrast to the percentile rank scpre 



where the score for the' total organization is at the 50th percentile. A 
unit could have a low percentile rank compared to the Department, but a high 
l^el of favorable responses. This could indicate that this management area 
IS not of immediate concern to the supervisor, * " 

Interpretation of the Scores of the MAS " 

Since the two scoring systems provide different kinds of ^information, the 
systems can be looked at individually, but provide more complete information 
when they are examined together. The percentile rank score provides compar- 
ative information on how a unit is Viewed by employees relative to all the 
units in the Department or agency. The percent favorable score shows the 
actual percentage of favorable responses made by employees in the organiza- 
tion. , 

In analyzing the two scores, the supervisor should look first at the percent 
of favorable responses. If the percentage, fb^: the unit is low in a given 
management area, for example, 30%, some attention should be devoted to that 
area by the supervisor. Even if the unit were somewhat abovk average on 
percentile rank, some improvement would still be needed i'l^he unit because 
of the low percentage of favorable responses. -/ 

On the other hand, if the percent favorable score for th^ unit is high (70% 
or higher)^ then a relatively low percentile score for the unit may not be 
a cause for concern. The overall level of achievement might be so high in . 
the Department that- a low percentile rank might be caused by the high overall 
standing of the Department rather than dissatisfaction within a particular 
work group. 

The. supervisor must review the results obtained, and decide which, if any, 
of the management , areas are in need of Attention. Whether any attention 
should be- devoted to a particular area would depend upon the score obtained, 
and the importance of the score area to overall effective performance in the 
unit. Only a few areas should be selected for improvement since changes in 
a large number of areas would be difficult to achieve within a reasonable 
time period... Generally speaking, those areas .,ia.which„the .Kork .group, has. . 
less than 50% favorable responses and where the unit falls below the Depart- 
ment average will be those most in need of attention". ^ 

Development of the MAS Scores 

The development of a highly sophisticated instrument such >s the MAS requires 
a number of relatively complex statistical procedures reviewed in this sec- ' 
tion. Initially the factor analysis results on the. item level data will be 
presented using individuals as the unit of observation. This analysis was 
used to map^out the score areas and to identify target 'items which became 
the subject of a second kind of statistical analysis--the item analysis pro- 
cedure--whu:h will also be discussed. As a result of the three separate 
item analyses which were carried'out to assign items to^^fJecific score 
areas, the 19 score arescs were finalized and are pres^/lted in this chapter. 
Following the 4)resentation of the score areas and tfy^r interrelationships, 
the next section in this chapter will describe the scoring procedure for the 



MAS. Tl^s will be followed by a presentation of the reliability of the MAS 
in terms of different reliability procedures. 



Factor Analysis of MAS ItetiTs 



As a result of the literature survey, continued discus3ion with DOL person- 
nel, and the pilot study results, 19 score areas were selected for measure- , 
ment by the MAS. Each score area was defined tentatively with five to seven 
items per score. A principle comj^onents analysis was carried out'on the item 
level data using the responses of 5,018 employees of the Department of Labor. 
Briefly siimmariz^ed, this analysis resulted in 14 factors that generally 
corresiJended to the a priori score area with four or more itenB defining each 
factor. The factors following were clearly identified. 



• Delegation of Authority 

• Supervisory Effectiveness ^ 



• EEO for Minority Groupf 

• Opportunity for Promotiohs 



Planning/Administrative Efficiency • Satisfaction- with Pay 



• Climate for Innovation 

• Work Satisfaction 

r 

• Performance Feedback 

• EEO for Women 



• Physical Working Conditions 

• Co- Worker Cooperation 

• Operational Efficiency 

• Workload Balance 



In view of the results found in previous studies, it. is particularly notewor- 
thy that the individually focused score of Work Satisfaction and Satisfaction 
with Pay were identified as separate factors independent from the other score 
areas. These results indicate that areas of traditional concern in attitude 
surveys can be identified separately from those factors concerned with or- 
ganizational climate and management procedures. 

The following score areas did. not emerge -as -separate ^£actor5..in.-,the. analysis^- 

• Upward Communication - ^ ^ 

• Morale 

• Fairness of Management 

• Trainring Effectiveness ^ 



U 9 



Downward Coitanunication 



The items defining these five score areas generally tended to load upon the 
Supervisory Effectiveness factor or upon another factor^ which was concerned 
with a particular source* of information, that is, higher level supervisors. 
Items .that were concerned with higher level supervisors or topi management 
which loaded on this one factor were found in the Upward Communication, 
Downward Communication, Morale,, and Fairness of Management score areas. 
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A review of the item content indicated that the factor could be well des- 
cribed as Employee-Management Interaction. • ^ 

A review of the data indicated that these originalt score areas could be . 
maintained through item analysis, and later more clearly defined bytlte 
development of new «r revised items. Sinde it was anticipated, that separate 
factors could be obtained for these score areas, and since having the score . 
areas Separate would simplify and clarify the feedback procedure by providing 
specific recoram^dations rather than diffuse global descriptions, it was ^ 
• decided to pursue the development of these areas as separate scores. Gen- 
erally, the item analysis procedures followed revealed relatively clearly 
defined score areas, especially for the Training Effectiveness score. 1 In 
the .principle components analysis, -"only two Training Effectiveness items 
were' included, since the remaining items in this score area had a nonlinear 
rekjonse 'alternative. In the cases of the other scor* areas, generally 
satisfacto# convergent and disr^riminant Validity results were obtained • 
through, the item, analysis procedures for the items in the separate score 

\ ■ ^ ■ ' . 
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areas ^ 




Descr4ptiQ!fi. of the Item Analysis' Procedures * 

The general procedure using item analysis techniques to develop) the final 
score areas involved selecting items within each score area which correlated 
higher with each other (convergent validity) than with items in Pther score 
areas '(discriminant validity). There were generally at; least two items in ^ 
each score area that met this standard. These items that met the desired 
standards of convergent and discriminant validity became target criteria tor 
an initial item analysis procedure. In this procedure, all items in the 
ms were then correlated with these initially defined target area scores. 
Items were then assigned to scores on the basis of the score area to »»hicb • 

• they were most highly related. Items which did not have patterns of conver- 
gent and discriminant validity sufficient to be assigned to a score area 
were then examined in a second item analysis run where the initial target 
score areas were expanded to include the items that had been identified in 
the first item analysis run. At the end of the third item analysis, ' final 

. - decisions -were- made on .the assignment, of. all . iXms, to, al l..score , areas 

An example, of the final result of this procedure is P^e^«"^«^/°!;^*^^,3'^jj;"^"2 
Effectiveness score in Table 2. In the upper part of the table are all the 
Training Effectiveness items and the correlations across the columns m^the^ 
table indicate their relationships above .40 to each of the f ^J^: -f^ 

will be noticed that the relationships of the training items to t^e training 
scores are substantially higher than their relationships to any other score 
areas, iTidicating the satisfactory nature of the convergent and discriminant 
validity analysis as developed in the item analysis procedure.. Although the 
training items and the correlations with the training 

'inflated since theV are part-whole correlations, the same pattern existed 
durijig the earlier^tem analysis runs when these were not part-whole rela- 
tionshiDs In the lower part of the table, starting with item 1, are indi- 
cated aU otJ^r^teiri; |h. MAS which correlated above -40 with the Training 
Effectiveness score and how these items were related to all of the MAb^ 
scores. These data were used to review all of the othe^ MAS -tems making 
sure that they were correctly classified and that they could not be more , 
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Tablb 2 

Item Test-Eta Coefficients for Training Effects 



ITEM 



M\S SCORES , 
11 



A 1 
U i 




X 


j04 


05 06 


07 
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45 






78 


41 






49'' 




40 


81 


45 


•4 U 




- 43 


42 


41 


76 


40 






46 


44 


41 ^ 


7S 


40 




40 


47 


45 


43 


76 


41 - 


J 7 




47 




45 


40 


53 • 


54 


43 


'^71 


40 


50 


50 


47 




81 


41 


45 , 


48 . 


49 


D J. 


43 


72 




49 40 


45 


51 






46 


40 ■ 


42 


49 


45 . 


44 




44 


41 


41 


40 


51 


C C 


45 


85 


45 


51 


48 


51 


50 


^55 




79 


44 


52 


H o 


42 


48 


40 


48 


46 


45 


bo 


4-i 






48 


43 


42 


cc 


42 


* 43 




54 


44 


43 


OO 




41 




4(5 


41 


48 


oy 


'to 


• o ^ 


40 


48 . . 


48 


48 


4 J 


40 


52 ■■ 




44 


43 


65 


/I 1 
44 


o o 


48^i 


j^'40 


44 * 


. 47 


42 


61 


45 




''^45 


49 ' ' 


52 


59 


56 


4 i 


A J 
4 *■» 




47 * 


47 V 


45. 


44 


40 


51 


77. 


, 40 


48 




47 


48 


47 




56 


41 


48 


53 


41 


82 


42 


46 


46 


75 


46 


56 




51 


42 ' 


44 


51 


50 


43 




75 


41 


46^ 


44 


45 


46 • 




79 


43 


49 


43 


42 


47 


69 




43 










. 71 




41 




40 


44 


48 






44 


40 


<9 


40 


44 


"44 


'40 


46 


41 


50' 




81 


49 


46 


4.9 


48 ^ 


54 


.46 


64 


40 


64 


50 


59 


S6 


48 


54 


46. 


53 41 ' 


49 


45 


50 


43 


56 




52 


51 


' 81 



12 



15 



44 



42 

47 
50 
79 
47 
43 



53 



41 

43 



40 

43 
46 
43 



48 
48 
49 
53 
48 

44 
59 
50 
SV 
75 
49 
53 
53 
77. 
'59 
. 58 
48 
61 
49 
53 
57 
58 
50 
46 
50 
50 
49 
46 
48 
41 
66 
77 
53 
53 
53 
55 



40 
40 
46 
44 



. 43 
48 
44 

~ 48' 
61 
44 
49. 
58 
53' 
83 
84 

' 53 
58 
48 
42 
47 
68 
41 
43 
: 44 
48 
58 
50 



49 
44 
54 
^3 
50 



*Only item-score etas a bove .40 with Training Effectiveness were used. 



appropriately assigned to a particular s,core area; This item analysis pro- 
cedure was followed for each of the MAS scotes before final decisions were 
made about assigning items to a particular score area. As a' result of this 
analysis, the final scoreareas described later in ^ this chapter were, developed 

An example of an item analysis printout with, location of various statistics' 
.indicated is presented in Table 3. These data resulting from item analysis 
procedures provide very comprehensive information 'about individual items 
and their relationships' to all .s^jf^" areas. Included in the analysis are 
ihfopraation concerning the numbel^f subjects", the percentage responding, 
and the percentage responding to each alternative, the item crite;^ion eta 
providing information about nonlinear relationships, biserial and point . 
biserial correlations for each alternative with each target criterion and 
other supplementary 'information which provides for a very thorough analysis 
of individual items. , , 

The end result of- this item analysis procedure was the development of the 19 
score areas included within the MAS. r 

Description of the MAS Score Areas . ^ 

%' 

The following descriptions of the MAS scores were taken fromf the MAS Handbook 
for Supervisors. These descriptions provide a^ definition of^.^cH score^afea 
and list the items that were inclucied.to assess each score area. yAlso^Msted 
are the S9ore areas that correlated most highly with each MAS score, mese 
correlations' were based on the intercorrelations among MAS scores and c^Tier 
variables across work groups with a total sample size of 1,261 work groups. 
Sipce these relationships .were based on larger samples than were available 
at the time the Handbook for Supervisors were developed, there ar^e some slight 
differences- in' these score areas listed. A complete table of score interre- 
lationships is presented in Appendix A which also includes a presentation of 
the intercorrelations among MAS scores across individuals- This later table 
is based on an N of 12,131.- 

1. Fairness , of Management is a measure of the degree to which management is 
perceived as fair and just in its treatment of employees (questions 13, 67, 
77, and 82). This measure includes the extent to which fairness, and not 
office politics and favoritism, governs promotions and other job functions 
(such as being selected for training, or having extended lunch hours, leave 
or other matters overlooked). It also included the extent to whidh credit 
is given to employees for work well done. As might be expected from the fac- 
tor analysis results, where Fairness of Management was ^ot a clearly defined 
factor, this score had a number of fairly high relationships with other MAS 
'scores.. It was most closely related to: Upward 'Communication (.69), Morale 
(.^7), Downward Communication (.65'), Supervisory Effectiveness (.61), Dele- 
gation of Authority (.59), Peformance Feedback (.59), and Co- Worker Coopera- 
tion (.59). It is particularly noteworthy that Fairness of Management is . 
viewed by employees as being a 4cey factor in influencing the description of 
a number of other SQOre areas, largely involving communication and morale. 
Certainly Fairness of 'Management kppears'to be a key factor in maintaining 
effective operational procedures from the employee's viewpoint. 

32 
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Table 3 

t 

Exaipfe of Item Analysis Printout Showing 
location 'of Various 'Smistics 



Number of subjects 
'not responding 
to this item -j — 



Item number 19 
Ci^iferion number - 
Item - Crit. Eta - 



• 7 



— i-.78 • 
Standard enor foreta ■> .00 



BLANK 



—.Number, of subjects choosing alternative coded 1 



v.\v 



-Percent of subjects choosing alteAiVe 1 

—Criterion mean for subjects, choosing 
alternative 1 




•Number of subjects chob'sing alternative 2 



5 , 



29' 'Onon .27'oJ4100 ,3H 2312 21U'l26 m 634 6% 
99.483 103.235 100.737" 97.688 - 96.479 96.073 

;00 ,0p .81 .60 .20 .16 -.55 -.65 -.38 -.63 -.32 
.-000-2- -f.013 3\ .012:2- .013 1 ..016 1 




Point biserial correlation between, " 
criterion and alternative 1 

• Key value for the alternative t 

— Standard error for the biserial correlation 

■Biserial correlation between criterion and alternative 1; 



i 
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2. Delegation of' Authority is a measure of employee perceptions of whether 
they have adequate authority to resolve work problems independently (questions 
39, 59, and 96). This measure includes the authority given employees to 

•plan theijr^^work independently arid decide how the work should be done. It also 
is a m^asfi^e of the extent to which lines of authority and responsibility 
are clearly defined within the unit. As would be expected, the Delegation of 
Authority score was most highly related to Climate for' Innovation (.6^), indi-* 
eating the importance of personal freedom and organizational support far inno- 
vative activi'ties. The secure was also- highly .related to otheVf factors involving 
personal commitment i such as Morale (.63) and an important component of Morale, 
Fairness of Management (.59), followed by other communication and organiza- 
tional process measures including Upward i-CommunicatiOn (. 59) ,• Downward Com- 
munication 0^57), Co-Worker Cooperation (.54), and Operational Efficiency (.54). 

3. Supervisory Effectiveness is a measure of employee perception of the 
general effectiveness of supervision within the unit (questions 20, 24, 43, 
7py and 90). It measures the quality of Supervisory review and follow-up 

of work done"^ within the' unit, the effectiveness with which work is organized > 
and scheduled, and how the supervisor aids in solving work problems. It^ 
is also a measure of the immediate supervisor's overall influence on the 
way people in the unit perform. This score had a wide variety of important 
relations with other MAS scores. It was most highly related to. Downward 
Communication (.63) and Operational Efficiency (.62) and was also related ' ^ 
to Fairness of Management (.61), Training Effectiveness (.59), and Perfor- 
mance Feedback (-.59). Since the supervisor is the center of interaction 
betweexi employees and management, the pattern of rela4:ionships obtained 
followed that which would be expected for this measure. . 

4. Planning and Administrative Efficiency is a measure of employee percep- 
tion of the overall effectiveness of planning* and the level of efficiency 
within a unit (questions 16, 68, 81, an()l.83). The measure includes the 
amount o© time spent doing unnecessary paperwork, the amount of time wasted 
due to poT)r planning, " the number of times poor planning necessitates changes 
in instructions after work has begun,* and the degree to which the rules and 
procedures of a unit facilitate or inhibit effective work performance. The 
Planning and Adminis^tratiye Efficiency measure, then, assesses ^the kind of 
planning structure which characterizes work group activity and, as such, it 
was most related to Downward Communication (.56), Training Effectiveness (.53), 
and Supervisory Effectiveness (.52). ^ - " 

5. Climate for Innovation is a measure of employee perception of the general 
level of emphasis on and openness to the development of new ideas and 
approaches to work problems (questions 44, 64, 78, and 79). The measure re- 
flects how often discussions are held on new ways of doing things, how often 
good ideas are sent up. to. the proper level for action, and how receptive the 
unit is toward new ideas and approaches. Climate for Innovation had a pat- 
tern of relationships similar to that of Delegation of Authority mentioned 
earlie'r, correlating' most highly with Delegation of Authority (.68) and with 
Upward Communication (.68), which in part includes receptivity to employee 
suggestions and ideas, as well as Morale (.61), Performance Feedback (.59), 
Co-Worker Cooperation (.58), and Operational Ef f iciency ( . 58) . 



6. Work Satisfaction is-a measure of the employees* reported general lev'el 
of satisfaction with their work (questions .10, 18, 27, 58, .and 73). It mea- 



siires the extent to which the employee finds his or her work bot;h interesting 
and enjoyable, how often the employees feel that the Kork offers the chance 
to accomplish something, and >iow muth overall satisfaction^ they gain from ' 
the work itself. The individually focused Work Satisfaction measure had a 
pattern of relationships- which integrated the concern of the individual and 
the organization. TH'S^core was ntost highly related to Morale (.59) and ^ 
Upward Communication 1*^9), indicating the extent to which the employee 
derived satisfacticm from involvement with the organization and acceptance* 
of the organization of the individual's ideas and contributions. Jt was 
also significantly^ Velated tp other scores which are of concern to the indi- 
vidual and the oi^ganization, i.e.. Opportunity for Promotion (.45) ,' Downward 
Communication (.45), and Delegation of Authority (.43). 

,7. Training Effectiveness measures employee perceptions of the effectiveness 
of training given to members of the unit as well as how well training needs 
are recognized (questions l9, '25 , 150, 60, and 94). The measure considers on- 
the-job training, training for new employees, training for new work methods 
and procedures, and the adequacy of the training received; 'This score was 
most closely related to: Downward Communication (.67yV,aup&rvisory Effec- 
tiMeness (.59), Upward Communication (.57), Morale (.57), and Planning and Admin 
istrative Efficiency (.53). Thus., the Training Effectiveness^RedTsure was 
related to variabiles which have a high component of st;:uctu)?e^and emphasis 
upon obtaining organizational goals, e.g., the relationsh^^ps with Downward 
Commiinication, Supervisory Effectiveness, etc. 

8. Performance Feedback is a measure .of' employee perceptions of the quanti- 
ty and quality of the work performance discussions held with employees in 
the unit (questions 38, 57, 47, and 99). As such, the ifieasure assesses the' 
number of times work performance is discussed with an employee; to what 
extent the information provided is useful and* yields practical suggestions 
for improvement; and how employees react to discussions of their work per- 
formance. As would be expected, this aspect of communication--Performance 
Feedback--was related to, other communication scores, specifically Downward 
Communication (.60), which measur^ the general flow of information downward 
to the employee, and Upward Communication (.60). It also had relationships 
to* a .variety of other scores as would be expected^ including Fairness of . 
Management (.59) and Supervisory Effectiveness (.59). 

9 . Equal Employment Opportunity for Women is a measure of employee percep- 
tion of the general effectiveness of the equal . opportunity program^ for 
women in the Department anjd the unit (questions 8, 28, 33, 45, and 69). The 
measure includes the extent to which women have the same opportunity as men 
to be hired and promoted to jobs within the unit, to receive training, to 
•become supervisors, a-nd the general level of success of the Department's , 
equal opportunity fok womep program. This Equal Employment Opportunity for 
Women score was basically a program evaluation measure which ^ as would be . 
expected, was mo^ related to Aether program evaluation measure for assessing- 
the effectiveriess'of the minorities program (.54) and to Fairness of Manage- 
ment (.37) as well as Upward and Downward Communications (.34 and .36, res- 
pectively) . 

10. Equal Employment Opportunity, for Minorities is a measure of employee 
perception of the general effectiveness of the equal opportunity programs 

. for mijiorities within the Department and the unit (questions 14, 41, 48, 75, 
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and 91). The measure considers the extent to which minoritie^/H^faVe the 
same opportunities as others, to be hired and promote^ to job^ within the 
unit, to receive training, to become supervisors, and the general level of 
success of the Department's equal opportunity for minorities, program. This 
management area was most closely related to: Equal Employment Opportunity^/ 
for Women , (.54), Upward Communication (*27), Fairness of Management (.26),'' 
and Downward Communication (.26). This program score on minorities was 
most highly related to the other program score on women with the next most 
highly related measures at a considerably lower level of relationship. 

11. Opportunity for Promotion is a measure of employee perception of the 
general level of promotion opportunity within the unit and the Department 
(questions 31, 47, 55, '70, and 88),. As such, the measure includes promo- 
tion opportunities in specific lines of work, opportunity for promotions 
within and from the work group, whether it is worthwhile to work hard for 
promotions, and whether the opportunities for promotions encourage the 

.employee to stay with the unit. This management area, was most closely 
related to: Morale (.66^, Upward Communication (.60), Downward Communica- 
tion (.51), Fairness of Management (.50). -Thus, 'the score £Crea on ^^^^^^^.^iii^ • 
Opportunity for Promotion seemed to be most closply associated with mea- 
sures that' involved the acceptance of individuals into the organization 
and the extent to which he feels involved with and recognized by ma^ge- 
ment. The correlations with Morale and the communication measured reflect . 
this orientation and these were followed by the Fairness of Management 
score, lalso an expected correla,te of an Opportunity for Promotion score.' 

12. Downward Communication is a measure of employee perception of the 
abilities of Supervisors at all levels to communicate successfully with 
employees at lower levels (questions 30, 46, 85, and 89). It measures whe-, 
ther employees know w&(it results are expected from work assignments, how 
often employees are Informed in advance of changes affecting the unit, -how 
often an explanation' of such a change is given, and how effective high level 
supervision is in keeping lower-level employees informed. The Downward 
Communication score was most highly related-to other scores which involved 

a communication component, specifically. Upward Communication (.73)^, Training 
Effectiveness C-67), Fairness of Management (.65), "and Supervisory Effective- 
ness (.63). " 

13. Upward Communication is a measure of employee perception of the degree 
to which management at all levels is interested in and aware of employee 
attitudes, problems, and ideas (questions 49, 51, 29, and 37). It measures 
the extent to_fcthich higher-level supervisors and top management are willing 
to have lower-level employees express their ideas about problems as well^ as 
their willingness to consider and discuss these ideas seriously. This,<f' - 
Upward Communication score had a somewhat different pattern of relationships 
than the Downward Communication measure, although the two were highly related 
to each other (.73). The Upward Communication score involved Fairness of. 
Management (.69), Climkte for Innovation (.68), and Morale (.68), which 
represented a somewhat different pattern of relationships from the other 
communication measures. 

14. Satisfaction With Pay is a measure of employee perception of the gene- 
ral adequacy of their pay (questions 5,. 26, 40:, 61, .and 84). The measure 
includes perceived adequacy of pay in relation to assigned work, compared to 
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other similar jobs, and to the local cost <of living. It is also a measure 
of how pay affects employees' attitudes toward their work. This Satisfac- 
tion with Pay score was the other measure which was largely focused on indi- 
vidual as opposed to organizational characteristics and it had a pattern 
of relationships with scores which were of concern to the individual, namely, 
Opportunity for Promotion (.38), Morale Cv34), and Work Satisfaction (.31). 

15. Morale is a measure of employe^ perception of the general state of 
morale in the work unit (questions 1, il, 93, and 97). Itjneasures the 
extent to which employees compare their work group favorably with other 
groups as a place to .worjc; see themselves as having a good future in the 
organization; and would stay with the unit if offered a similar job elsewhere 
at the same pay. This management ar^ was most closely rela.ted to: Upward 
Communication (.68), Fairness of Management (.67), Opportunity for Promotion 
(.66), Delegation of Aul^hority (.63), Downward Communication (.63), and. 
Climate for . Innovation (.61). Thus, the Morale score was of pervasive/ / 
importance in tems of a substantial number of variables which had high 
relationships. These relationships cut across organizational characteris-^ 
tics involving communication and fairness as well as characteristics impor- » 
tant to the individual, such as Opportunity for Promotion. . \ 

16. Jghysical Working Conditions and Equipment is g. measure of emplisoyee per- 
cepflon of the overall quality of physical working" space and equipment 
(questions 7, 21, 36, and 72). It is a measure of the quality of light, * 
heat, air equipment, supplies, work space, furniture, and cleanl.iness of 
restrogms and other facilities. This management area was most c-losely 
related to: Morale (.39), Opportunity for Promotion (.36), Climate for 
Innovation (.29), and Training Effectiveness (.29). ^ThuS,' J^^^^Ph'ysical • 

'working Conditions and Equipment score had a pattern of ffi^irly limited rela-. 
tionships with the other MAS scores, being most hi^ily related to measures 
which were of general concern to the individual. ' 

i' 

17. Co-Worker Cooperation is a measure of employee, perception of the level - ^ 
of cooperation among co-workers in their unit^(questions 2, 12, 34, 53, and 
71). It measures the amount of free information exchange between workers, 
the willingness of co-workers to assist each other in conyleting the work, 
their ability to work together to accomplish unit objectiyear, and the extent 
of any unpleasant disagreements /in the group. The Co -WojSl^ Cooperation 
measure was a key factor in theyOperational Efficiency meafflSre, correlating 
more highly with Operational^Bfficiency^ (.68) than any other measure. It 
was also related to other measures concerned with important organizational 
characteristics such as Fdimess of Management t*59). Climate for Innovation 
(.58), and Morale (.57). ^ ^ ^ 

18. Operational Efficiency is a measure of employee. perception of the work 
group's ability to produgje high level work effectively (questions 11, 17, 
62, 80, 86,. and 92). It measures how well the group handles difficult pro- 
jects; solves problems that arise; works under pressure; meets objectives; \ 
and readily adopts new, more effective approaches to problems. Opera- 
tional Efficiency measure -was most highly related to Co-Worker Cooperation 
(.68), followed by Supervisory Effectiveness {J>2) and Climate for Innova- 
tion (.58). "It is noteworthy that the Operat^Toaal. Efficiency Score had such 
a high relationship with Co-Worker Cooperatipn. ^This relationship generally 
supports the overall emphasis of the MAS on niiman rj^urces ahd their effec- 
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tive use. The other two correlates of this measure involved the center of 
organization operations--the supervisor--and its orientation toward improve-- 
ment.' .Overall, the Operational Efficiency measure appears to have a vevyj 
meaningful pattern ofl^relationships to other MAS scores, ' 



19, Wo:fllload Balance is a measure of employee perceptions of the degree to 
which the amount of work required of the wo^ group is- appropriate for the 
number of employees in the group; this includes whether there are too mjany 
.or too few employees to handle the heaviest .and the usual workloads, anjd 
*^>^hether there is too little.^or too much work to be done (questions 3, 23, 
42, and 74). Although^the Workload Balance score , had only limited relaltion- 
ships, the pattern was what might have been expected in terms of emplo>^ee 
views of effective matching of work to staff, as it cbrrel|ited most higjily 
with Morale (.26), Planning and Administrative Efficiency t-25), Satisfaction 
With Pay (.25), and Training Effectiveness (.24). 

■ -' ' ' / " ■ ,i« 

■ X \ r. ' \ ■ 
MAS Handbook for Supervisors '^i^-- 

— ^ ^ , , :^->^v^^. V . ■ ' 

The MAS Handbook for Supervisors is sent to ^sach sjupervisor .of five or more 
employees^ho receives -a computer printout ofj^e MAS results. The Handbook 
provides information concerning: ' -> 

1) • the two scoring methods; • 

2) the nature of the 19 score areas; 

3) the computer f eedba^ifci^^port ; and y2^.-^?:a^^ 

4) suggestions that can be used to counteract low scores. 

' Reliability of the MAS ■ . 

In the development of any psychometric instrument, an important consideration 
is the determination of the reliability of the measures. The analysis of. 
reliability may take different fcnrms depending on the nature of the instru- 
ment and the purposes for which- it was^eveloped . For the MAS, a minimum 
level of analysis was needed on the internal consistency of the score areas. 
For this kind of reliability analysis, the relationships among items de- 
fining a score area were examined to determine the consistency with which 
the area is measured. High internal consistency coefficients can be ob- 
tained either through highly related items, .or through the accumulation of 
a large number of items which result in a thorough samplin>> of the items 
within each score a^-ea. / , 

In either case, the intent is to obtain measures which would be representa- 
tive of what would be obtained if a, larger pool of similar items was used. . 
Internal consistency coefficients for the MAS were determined through the - 
calculation of alpha coefficients (Nunnally, 1970), and are presented in 
Table 4, In this table,' the average item relationships of the items in 
each score area a^e also presented. In view of the#limited number of items 
per score area, the alpha coef j^icients are highly, sei^tisfactory . Typically, 
internal consistency coefficients for most published\ instruments range in 

39 



^ Table 4 . 

Internal Consistency Coefficients and Mean Item 
Intercprrelations For MAS Scores ^ 





Internal Consistency 


Mean Item 


Score . 


LtUC X X ± ± CJ 1 U o 


T nt ercorre 1 at ion s 


1 Fairness of Management 


.78 


. - ' .47 


y2 Delegation of Authority 


' .70 


.37 


3 Supervisory Effectiveness 


.88 


.59 


4 Plan. § Admin. Efficiency 


.72 




-5 Climate' for Innovation ' 


.81 ° 


.51 


6 Work Satisfaction 


.89 


.62 


7 Training Effectiveness 


.88 


.60 


8 Performance Feedback ^ 


.76 ' 


.44 


9 EEO Women 


.86 


.56 


10 EEO Minority Grbups 


.88 


^ ' .-60 


11 Opportunity for Promotion 


.84 


^51 


12 Downward Communication ^ 


.74 ^ 


.42 


13 Upward Communication 


.78 , 


.4 / 


14 '.Satisfaction With Pay 


. 80 . 


.44 


15 Morale 


■ ^ ''|.78 ■ ]., 


.47 


16 Physical Working Conditions 


• '.63 \_ 


• * ' • .30 , 


17 Ca- Worker Cooperation 


.83 


.50 


18 Operational Efficiency 


.85 


.49 


19 Workload Balance 


.86 


.61 


i_ ! 
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the .80's and .90's, but such coefficients are ^obtamed with more items 
than are present in the MAS. V- 

Only one of the score areas in the MAS has an internal consistency coeffi- 
cient below .70--the Physical Work ing^ondit ions and ^Equipment score--and 
this is not surprising, since items dealing with facilities and equipment, 
lighting, etc. should not be highly related. The results can be summarized 
by indicating that ^he mean alpha coefficient of the MAS scales is .80 aiid 
that 11 of the 19 score areas had alpha coetficients of above .80. 

,The data screening procedure--SPECTR--developed to identify random responses 
and other forms of bad data also contributed to the reliability of the MAS 
scores. This procedure will be discussed more fully in a later section of 
this report. , 

Another form of reliability, which is crucial when the MAS is to be used for 
the smalysis of work group data, concerns inter-rater reliability. Jhat is, 
the agreement or the reliability of the observations of employees describing 
management procedures within a work group must also be high. Obviously, 
low levels of agreement between observers would indicate individualized ^ 
reactions to the management environment and would not identify problems 
which were pervasive to all employees within a unit. The intraclass correla- 
tion technique (Haggard, 1958) is'an excellent statistic for this purpose, 
even though the intraclass has been infrequently reported or used in the 
literature. The intraclass correlation coefficient provides aA indication 
of the inter- rater reliability of the average scores of the vrork groups 
studied, as' well as a measure of statistical significance of the results 
obtained. The magnitude of ^the intraclass correlations obtained are depen- 
dent upon the agreement of observers within work groups and the. differences 
between work groups in the scores on any given measure. With high agree- 
ment within groirls and large differences between work groups, higher intra- 
class correlatiofis result. 

Table '5 presents the average intraclass correlations obtained fc rhe 19 
score areas across six states and the Department of Labor. The high intra- 
class on Morale indicates that the members of a work group agree about the 
level of Morale present in that work group, and that the lev^l of morale 
would differ from one work group to another. The low intraclass correlation 
found on the. Satis fJttiqn^With Pay score indicates that there is some 
Qonsistency rr agreement within' work groups, but that this score cannot 
differentiate among work groups at the same level as the Morale score. 

Since comparable data on intraclass correlations in climate measures have 
not previously been reported, these results warrant additional discussion. 
For example, if a random distribution of sample means from wo«k groups were 
obtained where there were no consistent effects of management that were 
uniformly or consistently described by employees, the distribution of sampl% 
means by definition would approximate that of a normal chance distribution, 
with no significant differences beyond ch^ance for the work gVoup means. 
There would also be no significant relationships of the work group measures 
from the MAS with any type of organizational performance criteria. 

If only very slight levels of agreement were obtained or if differences 
between'work groups were only due to classifications of personnel on homo- 
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Table 5 



'Average and Range^of Intraclass Correlations 
^ for the 19 MAS Score Areas Across. Seven Organizations 





Score 


Average 


Range 



J 



1 


Fairness of Management 




.51 


.43-. 61 


2 


Delegation of Authority 




0 

.56 


.40-. 71 


, 3 


Supervisory Effectiveness 




« 

.63 


.55-. 67 


4 


Plan. § Admin. Efficiency 




.60 


* .45-'.70 


5 


Climate for Innovation 




.58 


.35-. 75 


6 


Work Satisfaction 




.44 


.15-. 63 


7 


Training Effectiveness 




J .51 


.30-. 60 


8 


Performance Feedback 




.43 


.27-. 56 


9 


EEO Women 




.47 


.30-. 65 


10 


EEO Minority Groups 


* 


.33 


.10-. 49 


11 


Opportunity for Promotion 




.48 


.4 3-. 56 


12 


Downward Communication 




.45 


.30-. 57 




Upward Communication 




c n 
. b / 


.4o- . OD 


14 


Satisfaction With Pay 




.47 % 


.32-. 63 


15 


Morale 




.68 


.60-. 75 


16 


Ph/sical Working Conditions 




.72 


.58-. 80 


17 


Co-Worker Cooperation 




.66 


.56-. 73 


18 


Operational Efficiency 




.51 


.07-. 62 


19 


•Workload Balance 




.63 


.45-. 76 
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geneous individual differences, then large categories of data such as males, 
females, older workers, certain personnel classifications, etc., would be 
needed to obtaip meaningful and significant relationships with other kinds 
of variables. This is the typical way in which attitude, data are anialyzed. 
However, with significant intraclass relationships as found in Table 5, 
results can be reliably reported on individual work groups Hrhich do result 
in highly significant statistical differences. The use of the intraclass 
correlations to assess work group agreement and differences between w(5rk 
group's thus ptovides a very important assessment tool, and demonstrates a 
form of reliability and consistency which has not characterized previous 
research. . £> ' 



The Validity of the MAS 

The ultimate purpose of the Management Audit Survey is to improve the use 
of our nation's human resources. Such an abstract statement of the long- 
term goal of this research has a number of components when analyzed by 
operational definitions. Initially, and historically, such studies can be 
thought of as looking at the analy^s of work attitudes and satisfaction by 
individual descriptions and .reactions to standard questionnaire items, 
which are then used as indicators or t^tative indices to provide information 
about other more distant and abstractly measured criteria, e.g., the satis- 
faction or dehumahization of the work force in a period of rapid social 
change. ^ 

This level of analysis is concerned with individual perceptions of 'satis- 
faction with organizational procedures and the primary focus of analysis is 
upon the individual and his responses. Alternately, such research can be 
viewed through an organizational perspective. At this level of analysis, 
work groups become the crucial component and individual responses are 
averaged, removing the effects of idiosyncratic perceptions from the analysis. • 
If the ultimate objective^ of- such research is to bring -about -higher levels 
of organizational perfprmance, and work satisfaction, then MAS variables must 
be analyzed through wprl^ group data where the relationship of MAS measures 
to organizational performance criteria and average levels of sat i section 
can be e^^mined. Thus, throughout the deyelopmental sequences of "the MAS, 
obtainingN^ccess to organizational criteria was an important and necessary 
'step l^fore the MAS could actually be evaluated on the long-term goal of con- 
tributing to more effective human resource management as well as being useful 
at the individual/^level of analysis.. 

When the opportunity to study the Employment Service and Unemployment Insurance 
operations in various states arose, a variety of possible performance measures 
on these organizations became available. Unfortunately, there was a wealth of 
information about local office performance, but no systematic treatment of 
the nature of organizational performance criteria--their limitations of relia- 
bility, relevance, redundancy, predictablity, etc., and other standards^ for ^ 
performance criteria were not known. 

A whole host of measures <wer^ .collected and reported on local 9ffice opera- 
tions included in the Employment Service Automated Reporting System (ESARS) . 
Essentially, this procedure collected very extensive information of the type 
of persoiv applying for services, a record of all processes or activities 



carried on yith the applicant while he was being seorved or not searved, and 
finally processed or eliminated from ES rolls. Discussions with Employment * 
Seirvice personnel about available data provided generally discouraging in- 
formation about their reliability, relevance, and contamination withsitua-' 
tional effects, etc.; in short, their general utility as indices of either 
local office performance measures or as criteria which could be used to 
monitor actual performance and bring about higher levels of Employment Ser- 
vice performance was at least limited. 

Ideally, of course, performance criteria c^ serve '^as crucial guides for 
the development of any organization in monitoring its development, evalua- 
ting achievement, diagnosing deficiencies or opportunities for improvdbent , 
providing feedback about process and outcome measures and integrating ^these 
concerns for decision-making at all levels of 4he organization. In the long 
run, research on performance criteria should be integrated with research on 
human resource management in a systems perspective with other management 
ijiformation system data to present information of value to each level of the 
organization in a timely and clearly understood fashion, to bring about 
Higher levels of performance. 

The available data base for an examination pj^ the relationship between MAS 
scores and various Emplo>Tnent Service performance criteria was obtained in 
the State of Pennsylvania, where data on 78 local offices w^re available. 
In these 78 local offices, the number of MAS respondents varied, depending 
upon the office,- but only offices with three or more people were considered 
in the analysis. There were a total of 963 employees in these local offices. 
Data were also available on 539 employees in .73 Unemployment Insurance 
offices, and 123 employees in 22 employment offices in the State of Missouri. 

The procedure followed in carrying out the validation study of the MAS ^ 
involved computing average scores on the MAS for each local office— a routine 
procedure in the processing of MAS data— developing criterion measures for 

each "l o cal 6Ff i^^^^^^ sets^ o f measures - ;Because- of- the 

lack of previous research, a great number of possible criterion measures 
were examined. Because of the extensive nature of these data, only a small 
portion of the criterion study results will be presented here, that portion 
that appears to have the most relevance for future research and Employment 
Service action. Table 6 presents the validity coefficients of the MAS scores 
with selected performance criteria. 

These results in^cate that the percent of first payments of Unemployment 
Insurance completed on a timely basis, according to the state criterion, was 
predictable by a wide variety of MAS scores. Significant z^ro-order validi- 
ties were obtained for 14 of • the 19 MAS scQres--e.g. , Operational Efficiency, 
(r = .47), Training Effectiveness (r = .37), Fairness of Management (r = .34), 
etc. This criterion is largely one involving completion and processing of 
paper work, according to administrative procedures. Thus, there appears to 
be some indication of construct validity in the nature of the MAS-performance 
measure relationships. 

The two criteria, obtained from Missouri were the intake cost per applicant 
and the employment cost per applicant. Both of these measures indicated 
the costs assigned to this function by the accounting system. Since these 
criteria were stated in cost results, relationships with the MAS scores 



Table 6 || 

Validation oi MAS Scores for Different Orga 
Performance Criteria and Geographical U> 



MAS Score 



I. Faifness of Management 

2 Delegation of Authority 

3. Supervisory Effectiveness 

4. Planning § Admin. Effic. 

5. .Climate for Innovation 

6. Work Satisfaction 

7. Training Effectiveness ' 

8. Performance Feedback 

9. EEO Women 

10. EEO Minority Groups f~" 

II. Opportunity for Promotiyns 

12. Downward Communication 

13. Upward Communication 

14. Satisfaction with Pay 

15. Morale ^ 

16. .Physical Working Cond. § Equip. 

17. Co-Worker Cooperation 

18. Operational Effic^ncy 

19. Workload Balance 



.34** 

.34** 

.34** 

.2^** 

.32** 

.13 

.37** 

.22* - 

.36** 

.12 

.15 

.14' 

.28* 

.20 

.34** 

.24* 

.33** 

.47** 

.28* 



-.44* 

-.37 
-.37 

.03 
-.36 
-.23 
-.41* 
-.38 
-.25 
-.22 
-.31 
-.32 
-.27 

.11 
-.32 

-.03 
-.23 
-.47* 

.13 



is 
-.10 
-.21 

.11 
-.28 
-.24 
-.18 
-.46* 

.13 . 
-.11 
-.18 
-.18 
-.04 

.31 
-.02 

.02 

.04 
-.39* 

.24 



^Unemployment Insurance Offices, Eastern State, N = 73. 
^Employmwit Service Offices, Midwestern State, N =» 22. 
^Employmerit Service Offices, Eastern State, N « 78. 

< .05 
**£ < .01 V 



should bfe negative, that is., high MAS scores should be associated with low 
cost per unit of output?.. The results indicated that the expected pattern 
was obtained. For the intake cost -per -applicant, the key scores, which had 
simificant validity .coefficients, were Operational Efficiency and Training. 
EfScti^enesI ' The int^lce process is, again, a clerical, procedural pro- 
cess, and the sajjie two measures had significant validities against a some- 
what similar criterion in a different geographical location in a different 
kind of.organizatiop. Essentially all of the MAS .scores had negative rela- 
tionships with this criterion measure, as was. expected. 

On Employment Cost per Applicant, two scores had. significant relationships. 
These were Operational Efficiency (r = -.39) and the Performance Feedback 
score fr = -.46). Again, the, majority of the MAS scores h^ negative validi-. 
ties as expected against this criterion that involved' the number of place- 
ments and the total costs for that placement. 

■ The next criterion listed in Table 6, Job Development Placements per Appli- 
cant Available, was a kind.of innovation measure, involving the number ot 
. job openings ci;eated as a percentage of the total applicants served. This 
criterion 5as predicted at the .05 level of significance or beyond by three 
MAS scores: Climate for Innovation (r = .41). Co-Worker Cooperation (r = 
.32), and Operational Efficiency (r = .24). The innovation score in the 
MAS was the most highly related to this innovation measurer, another indica- 
tion of the construct validity of the MAS. 

The nexJt criterion measure. Referrals per Placement, was a process measure 
dealing with efficiency, involving the number of referrals made per pUce- 
ment. This criterion was predicted significantly by 10 MAS scores., Sxtite 
, this measure is an indication of the relative inefficiency "^cessary^to ob- 
tain a' placement, MAS scores should, have a negative relationship to this 
score i e. when a high number of referrals had to be made to obtain a 
placement, one. would expect that management procedures were relatively 
inefficient in the lis e of human resources. The scores with significant 
relationships ^'with this^ criterion were Planning and Administrative Effi- 
clenc^Cr = ^37)/Co-Work5^,Cooperatio^ (r = -.33), Operational Efficiency 

(r = - . 33) , etc . 

The next criterion. Placements per Applicant, was a percerl^of potential 
measure, involving the number of individuals placed as a Percentage of ■ 
applicants available. It was predicted at the 05 level °%beyond by 13 
of the 19 MAS scores, including Performance Feedback (r = .38) Downward 
. communication (r =. 33) , .Upward Communicati*« (r = '^^^ ' ^ork^^atisfaction 
fr = .31), Planning and Administrative Effic!\cy (r = .29), etc. Since 
this "criinrion measure is a key one for Employlfent Service operations, that 
is, the percentage o5 clients served, the number of MAS scores that correlate 
^gnificantly wi?h it is a further indication of the ^^-"PO^^^"" ^^2^ 
sSlts The most valid score for this criterion. Performance Feedback, was 

• also the most valid sci^re in- predicting the similar f "^^^^^f"^";. 
Employment Cost per Applicant. These results ^"gg«f ^J^^^. ^O tSs as- ' 

■ in obtaining a high number of placements is frequent attention to this as 
pect of Employment Service tjperations. 

The final criterion listed in Table 6 is Service per ApPji^ant, which 
involved the number of applicants placed, tested, counseled, and trained 
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expressed as a percentage of the number of applicants available. This 
measure was predicted by eight of the MAS scores. The most valid MAS 
score was Operational Efficiency (r = .36), followed by the Co-Worker 
Cooperation score '(r = .32). ' 



These data can be isummarized briefly by saying that the MAS was signifi- 
cantly related to a number of different performance criteria in different 
geographical locations and that a meaningfully consistent pattern* of rela- 
tionships existed between the MAS scores and the different performance 
measures. 



Data Processing Screens 

An importa^^t part of the MAS system, which has contributed to. the results 
described above, was the application of certain scoring procedures designed^ 
to select those MAS data which were reported in an accurate .and sincere 
manner. Prior to application of the SPECTR screens described below, the 
code numbers reported on the answer sheets were checked for accuracy af 
only 1 or 2 individuals used the same organizatiqnal code. Respondents 
with code numbers which did not appear in the code book were deleted from 
the data bank. 

With any survey endeavor of the MAS- type, a certain amount of erroneous 
data is returned by the participants. 'This may come from several sources, 
such as participants responding-n!n a random fashion, responding to most or 
all questions with a particular set, answering only some of the questions, 
marking the same response alternative to the items losing the place on the 
answer sheet, etc.. With most surveys, many errors of this type are included 
in the analysis. However, within certain statistical probabilities, much of 
the more blatant forms of erroneous data may be eliminated. 

SPECTR is a computer program developed by IBRIC to overcome the error that 
results from response sets, random responding, etc./* This program screens 
out all respondents who answer the MAS in obvious ly^iased ways, and does 
not allow their deviations to influence the results. The percentage of 
answer sheets typically rejected by the various screens has been about 10%. 
The amount of error represented in this percentage would obviously have made 
a tremendous difference in the interpretation of results and in the relia- 
bility of measurement. ' , 

The SPECTR screening procedures involved the following parameters: 

1) Number of illegal responses--the answer sheet\ised for obtaining response 
to tiie MAS was designed to accommodate questions with five alternatives. How 
ever, certain questions in the MAS had fewer than five alternatives,^ This 
allowed the respondent to mark a response on the sheet which did not appear 
in the questionnaire. Such a response was called an illegal response. A 
large number of illegal responses could indicate that the respondent was 

not attending to the questionnaire, was not marking responses in the approp- 
riate area of the answer sheet, was deliberately making erroneous or random 
responses, etc. 

2) Number of missing responses--this is simply the number of items not 
responded to, or left blank. Illegal responses were also counted as missing 
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respon?es. It^was felt that a large number chfOnissing responses indicated 
an unwillingness to coopQrate, inadequate time ro answer all questions, ^etc. , 
and the resultant area scores would not adequately reflect the organization's 
level on the dimension measured. 

3) Wi thin-Score Consistency (WSG)'--given a set of score areas, each con- 
taining relatively homogeneous items, a score can be computed for each 
participant based on tihe avei:age variance among the items in each score 
area. This average within-score variance for the set of 19 MAS score areas 

'^^ is called the Within-Score Consistency. A large WSC sc^ore indicated that 
the respondent did not tend to respond similarly to items wl>ich had similar 
content and may have been marking responses on the answer sheet in a random 
fashion. An extremely small score would also indicate that the respondent 
was not respoiTjding meaningfully to the items that made up a score area and 

* was deliberat^y answering to the same degree to each item C^.g.; all posi- 
tive or all n%ative); in short, there was very limited variability in the 
employee's responses. Cutting scores for each end of this distribution 
were set so that' approximately 5% of the subjects were rejected for random 
response patte^ms, and less than 1% of the. subjects were rejected for rigid 
response patt4r7|5. As a check of the effectiveness of these computer screens, 
50 answer shej|t^were filled with random responses and processed. All 50 
of the answerfshfeets were rejected by SPECTR. Twenty-two were rejected by 
the illegal r|sponse screen, which was the first applied; the remaining 28 
cases were rejected by the random response screen. Thus, the reliability 
of these procedures was very good. 

4) Across Score Consistency (ASC)- -extremely favorable or unfavorable res- 
ponse patterns resulted when subjects attempted. to portray the working en- 
vironment either in an excessively positive or excessively negative manner. 
These screens were based oh the rationale that, if a number of scores were 
not correlated, it was extremely unlikely that a person would obtain a con- 
sistent pattern of very positive or very negative responses when a mdre ran- 
dom patteniKwpuld be expected. The situation is somewhat analogous to ob- 
taining all -heads in tossing ten unbiased coins, where the probability is 
the product /Of the individual probabilities.' A sample of approximately 
5,000 case3 was used to determine the relationships among scores. Nine 
pairs of variables were then selected which had near-zero correlations. For 
each nearl/zero r, an individual's score for one variable was multiplied by 
the score ion. the/otTTer variable; these were then summed so that an average 
cross-product score could be obtained for the variable pairs seleclped. Those 
who had excessively high (unfavorable response set) or excessively low (favor- 
able response set) average cross-product scores could thus be screened out. 
While exact probabilities were not computed becausQ the correlations were, 

in fact, not zero, it is not very likely that many satisfactory or^ legitimate 
cases were rejected." In the Department- of Labor data, less than 3% of the 
cases were rejected for a ^favorable response pattern, and less than l%*for. 
an unfavorable Response pattern. 

^All of these screens helped to ensure that the resulting data concerning the 
characteristics and operating procedures of the individual units were more 
accurate. These screens represent an unusually sophisticated analysis, 
which ds not yet available on any other similar management system. 
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Summary 

The unique properties of the MAS allow it to be used as a separate system 
or to be* combined with other organizational audit and development functions 
to give a more, thorough and in-depth analysis of the situation in any par- * 
ticular agency or organization. The MAS offers an excellent system for 
analyzing the human resource management practices of an organization, or, 
on a larger scale, it can assess the current organizational climate and 
offer recommendations for correcting problem areas. 

The features of the MAS that, can be stressed would include: 

1) Each supervisor of fiver or more employees receives a computer feed- 
back' report describing the state of human resource management for that 
work group across 19 score areas. Higher levels of management r^belve 
individual feedback reports, copies of reports received by suborcfinate 
supervisors, and a combined repart for all employees subordinate to him. 

2) The MAS was designed with' an underlying behavioral emphasis* This em- ^ 
phasis facilitates the development of programs for cons^uctive change as 
♦the ^luestionnaire itself contains descriptions of thosie activities* which. 

can lead to improvement of a {i'|trticular management procedure. This beh 
vioral emphasis arlso. contributes to high agreement amonjg employees withi 
a work group. 

3) The system offers two kinds of scoring information--a percentile rank 
score, and a criterion-referenced scox,e--so that both comparative and ab- 
solute information are available to facilitate effective interpretation 
of the results, 

4) The MAS Handbook -for Supervisors provides direct, behavioral ly-based 
suggestions for organizational development for each of the 19 ^core areas 
and information which will aid in the interpretation of the scores. 

5) The system uses a sophisticated computer program to screen^ out variou^ 
response patterns and thus increase the reliability of thje scores fed back 
to each work unit. The response patterns excluded by the program include 
random response^ patterns, excessively positive or negative patterns, re- 
peated response patterns, and excessive missing and/or illegal responses, 

6) The system has unusual reliability for its length, as all scores except 
one have internal consistency coefficients of ,70 or better, furthermore, 
the MAS uses intraclass correlations to assess agreement between obsei^ers 
where highly significant results have been obtained on all 19 scores, 

7) Although relevant performance criteria for this kind of instr^ent are 
difficult to obtain, significant validity .coefficients have been obtained 
to indicate that the MAS can predict organizational performance measures ^ 
relevant to human resource management. To illustrate, a variety of organi- 
zational performance criteria concerning the operation of Employment Service 
offices in two. states, and a productivity measure from Unemployment^^nsurance 
offices have been predicted' significantly with a variety of MAS scores! ' 
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:) Tn:- MAS has also shouTi a pattern of significant relationships with 
vai;iou5 control scores such as office size and -workload. These results 
provide guidelines for certain aspects of human resource management that 
can be followed* to overcome the situational variables which impact on the 
organizational performance measures. 

9) Extensive norm data have been accumulated on over 20,000 employees. 
The return rate for the instrument has been surprisingly high, generally, 
greater than 90%. . ' 

10) Specific training deficiencies can ^e identified and treatment provided 
only to those areas of the organization in need, thus making for more effec- 
tive* and economical organizational development.^ The use of the MAS in train- 
ing programs should be highly relevant since problems of transferring course 
content to the work situation' should be essentially eliminated as training 
data can be based on the trainee's own work group. ^ 

In view of the significant results obtained, further research* and applica- 
tion of the system were warranted. The^ current research effort was concerned 
with the generalization of previous results and with the development of the 
system in a study of management procedures, ES performance, and a number of 
control scores in six states. 




chapter iv 
Vr'ocedure 



Th^ 'anihter will fpresW^the, procedures followed durij)&^||^e search^ 

...^1. ^ ji^^^^A^i^A-^^ ^4^1 oontT^io The saip^3?^a^^^l?n will in- 



ij^iiuilig with a description of ' the sample. _ 
^.li^e^escriptions of the data collection procedure:s,,^|S^number of indi- 
viduals completing; the Management Audit Survey, ^and the number of work 
groups included in the analysis of the criteria, control scores, and MAS 
data. cThis section will be followed by a discussion , of the criterion pro- 
blem-, which, as in any research of this kind, is of paramount importance. 
A dei^i^tion of the control scores considered in the study and how they 
were artalVzed to examine^ their impact on 96 performance criteria^wi 11 be 
discussed next. Finally, the chapter will conclude with a description 
of the data analysis procedures used to examine the interrelationships 
of the^MAS, criteria, and control scores. 

Description of the Sample 

Tills MAS data were collected through the efforts of the Audit and Inves- 
tigation regional representatives. They and the monitors from Washington 
visited the participating states, and arranged for the administration of 

.the MAS. The states included in the study were selected on the basis of - 
their general representativeness, willingness to coo^^erate, and the judged 
quality of their criterion data. The participating states of Missouri, 
North Carolina, and Tennessee had been studied with the MAS in June of 
1974. New data collection was carried out in the. states of Ohio and Texas 
in October of 1975 and Mississippi was added in December of 1975. This ^ 
array of different time periods of data collection provided an additj.onal 
benefit to the study, as different economic conditions were in effect at 

\the time of the data collection, providing an opportunity to assess the 
impact of the control scores under varying economic and geographic conditions, 

A description of tide number of subjects included in the study from each \ 
of the participa;t^ng states is presented in Table 7. 'The table shows 
estimates of the total number of employees from each of the states, the 
number of answer sheets scanned for each state, and the resulting parti- 
cipation rate u^i?t the MAS. The overall average' return rate for the six 
states studi?^' was 91%, a remarkable participation rate for studies of this 
kind.^ The tabj^al so presents the number of^subjects that^were dropped 
from the stucfy^ the specially-developed computer program SPECTR before 
average scores for .each work group were computed^ Inspection of this 
table indicates. that the random response screen was responsible for de- 
leting the largest number of answer sheets. At the bottom of the table, 
the total number of subjects deleted from each state is presented, tog,ether 
with the number of subjects, that were used for the computation of the MAS 
results. Finally, the table f)resents the total percentage of subjects re- 
jected. This percentage ranged from a low of 5% in North Carolina to« a 



Table 7 

Number of Subjects Rejected from the Sample for 
Each State and the Total Sample 



MS' MO NC OH TN TX . TOTAL ^ 



r 

Population Estimate 


920 


2000 


1588 


3200 


1301 


3900 - 


12909' 


Sheets Scanned 


9a9 


1795' 


1387 


2899 


1129 


3748 


11867 


% ^Return 


99 


90 


87 


91 


87 


96- 




SPKCTR Screens: 

Unfavorable I^esponses 


1 
1 


■ 7 


7 


38 


5 


25 


83 


Favorable Responses 


6 


34 


12 


32 


29 


119 


232 


' Random Responses 


26 


76 


36 


145 


48 


193 


524 


Rigid Responses (WSC) 


9 ' 


11 


4 . 


11 

* 


8 


21 


64' 


Missing and Illegal 


13 


4 


4 


50 


6 


39 


116 


Bad Code Number 
Total Dropped 


5 


24 


12 


14 . 


5 


88 


148 


60 


156 


75 


290 


. 101 


485 


'll67 


% Rejected 


7 


9 


. 5 


10 


9 


■ 13 


10 


Feedback N 


849 


1639 


1312 


2609 


1028 


3263 


10700 



/ 



53 
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high of 13% in Texas, with an overall average of 10% for the total sample. 
The availability of a sophisticated computer program such as SPECTR to 
^eliminate bad data is obviously an important benefit to the study. . 

Since the primary purpose of the study, was to examine the relationships 
betw^een various performance measures -aTid management procedures as assessed 
by the MAS, the local offices included in the study had to bp comparable 
in their opportunities for criterion performance. Thus, from the^ total 
sample presented in the previous table, all personnel not in ES offices 
were screened out, and the remaining sample of local offices was further 
screened on other characteristics. A list, of these screens and the number 
of offices eliminated by each are presented in Table 8; Several of the 
screens listed on the table overlap, and many officJes would have been 
screened out for more tiian one reason/ Screens 1 or 2 were applied to 
those offices identified as missing either MAS^or ESARS criterion data. 
Screens 3- and 4 were designed to eliminate the* very small or very large 
offices. Since the MAS was designed to. foctis primarily on the management 
problems of each work group throughout an organization, extremely large 
work groups, which were not representative because of their size, or 
extremely small groups, where there were not sufficient observers to 
obtain stable estimates, were screened out. The Sth screen, on type of 
office, eliminated offices where criterion data were not directly compar- 
able to the bulk of the total sample. Screens 6 and 7 were very similar 
to screen 3 which eliminated very 'small of fices where the criterion data 
were less likely to be reliable. The Sth screen applied only to Mississippi, 
where there were'a number of local offices witlh.MAS data available, but. 
ESARS data for these offices had been combined and were not available sep- 
arately. * When the ESARS measuVes for more. than three local offices had> 
been combined , .those offices were eliminated from the Mississippi sample. 
While the remaining offices in the combined sample represent the vast 
majority of ES offices in these states, %nd should provide some .general 
applicability, for criterion findings ai)d MAS relationships, the removal 

of different kittdfe; of offices dQes -limit the generality o£ the- findings-- 

of the. study to similar kinds of offices. 

. . V ; The. -Criterio^^^^^^^ 

The definition and quant if icatibn of .performance goals, objectives, and 
standards for individuals and organizations has traditionally been labeled 
the criterion problem. This is ,a subject, which ha^ generated much discussioTi 
but limited research,, and even less consensus about how implicit and explicit 
issues should be resolved. ' 

Despite the lack of consensus aAong investigators^conceming the criterion 
■'problem, one facet of agreement can be found--its paramount importance-. 
Performance criteria serve as the crucial guides for the development of 
any social program or organization in defining the goals of the organiza- 
tion, monitoring their development, evaluating achievements, diagnosing 
deficiencies or opportunities for improvement, providing feedback about 
process and outcome measures, and integrating these concerns for decision- 
making at all levels of the organization. One of the more interesting - 
and recent treatments of the criterion problem, which reviews a number of 
issues inherent in the area^ is an article by Ja]i^(1973) . He reviewed 

* ' .. 
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Table 8 



Number of Local Offices Eliminated from the Sample for 
Each State and the Total Sample 



■ 


MS 


MO 


NC 


OH 


TN 


TX 


TOTAL 
















\ 


T- ^ 
















Total Number of Offices 


36 


70 


11 


1 lo 


ol 






Screens: 
















1 . No ,,MAS Data 


0' 


■ 11 


6 


31 


0 


10 


58 


.2. No ESARS Data 


0 


' 3 


4 


\ 4 


>^ 


9 


23 














48 


/^J^^^^S N Less Than 3- 


6 


12 


2 


llS 


^2 


15 


4. MAS N More T> an 50 


0 


2 


0 


2 


4 




10 


' 5. WIN, CEP, CETA, Job Bank, 
















Qii a 1 T aHnT Tl" i fi ptjiti 1" 
















Rural Manpower^ ' 


0 


29 


14 


.?4 


1 


35 


113 


6. Applicants Available 
















Less Than 100 f , . 


•0 


0 


5 


2 


0 


1 . 


. 8 


7. Placed Less Than^SO • 3 




% 










36 


0 


14 


5 


4 


0 


13 


8. More Than Tb^e Offices ^ 
















Combined oilESARS Data . 


8 


0 


0 


0 


0 


0 


8 


Total Offices Screened Out 


.14 


38 


17 


53 


11. ■ 


53; 


186^ 


Remaining Offices 


22 


32 




6.5 


20 


' 71 


265 ^ 



0 



various criterion models, including: 1) the ultimate criterion model, 
which argues for general measures or an overall composite; 2) the multiple 
criterion model', which argues for separate treatments for separate criteria 
and 30 the general criterion 'model , which adopts a systems perspective in 
its developments-including individual differences, task demands, organiza- 
tional reward structures, and training and development experiences--which 
leads to organizational outcomes that are interrelated to the inputs and 
process measures through feedback loops. These approaches are not neces- 
sarily "mutually exclusive and James argued that a necessary consideration 
for criterion development is construct validatian procedures where measures 
or indicators are combined through theory and empirical data. 

The ES performance measures examined in tfie study can be foi^d in Table 9. 
These measures were selected on the basis of a review of the results ob- 
tained in the Pennsylvania study and from measures developed, or suggested, 
by other contractors. 'There are a number of comments that are appropriate 
concerning this table. To begin with, the list itself is quite comprehen- 
sive, covering a wide variety of different kinds of performances which ES 
offices carry out. The table considers functions pertaining to applicants, 
employers, to special target groups, special services given to applicants, 
as well as a number of internal effectiveness ratios and two measures of 
quality of placements. 

The measures were also classified by type of criterion measure. These 
are indicated by the parenthetical letters which follow the individual 
Criterion measures. The- three types were as follows: (A) measured of 
output in relation to rost;^ (B) level of performa^ice me'asures, such as 
the percent of applicants placed; and (C) penetration measures with which 
the performance of the local office was compared to the total market 
activity, such as the number of applicants available expressed as a pro- 
portion of thte t tal unemployed vvithin the county served by the lo9al 
office. \ 

The fact that all of these measures involve some kind of ratio witrli shared 
terms ^makes their analysis according ^to methodological aSid psycjiometric 
standards more difficult. These standards of sensitivity, redundancy, 
reliability, con:;truct validity, con1>GJnination, etc., will be discussed 
when the criterion relationship^ are presented. The sharing of common 
terms in numerator an^Voi denommator also made difficult the use of 
certain statistical tecluiiques which have been widel)^ used to examine 
criterion relationships (e.g., factor analysis). The data, howeVfer, 
could be analyzed for reliability for four of the six states, where 
quarterly data wegre obtained. These states were Mississippip->North 
Carolina, Tennessee, and Texas. Yearly data only were obtained for 
Missouri and Ohio and, thus, an analysis of their reliability through 
correlations of datafrom adjacent quarters could not be carried out. 

An important consideration in the analysiS;^of criterion data is the 
extent to which they were contaminated by ^:ituational variables, or, 
as they are more typically called in the psychometric literature, con- 
trol variables--variables which have an impact on either the criteria or 
the predictors' and make unequivocal interpretations of the results dif- 
ficult. In the analysis of ES functioi>s, these control variables were 

^ . ■ ^ 56 ■ . ' 



Table 9 

*• Examples of Criterion Categories 

an'd Individual ES Performance Measures^ 



I. Labor Exchange Function 



Applicant 'Service Function 

1. Individuals Placed / Applicants Available (B) 

2. Individuals Placed without Training / Applicants Available (B) 

3. Applicants Available / Total Unemployed (C) 

4. Individuals Placed / Total Employed (C) ^ 

Employer Service Function 

5. Openings Filled / Openings Listed (B) 

6. Openings Li'Sted / Number of Employer Contacts (B) 

7. Openings Listed /-Total Employed (C) * " , 



? II. Extended Services 



Special Tairget Groups • ' 

8'. Composite Measure (s) of the follpwing ratios: (B) 

Veterans Placed/Veterans Listed 
Minorities Placed/Minorities Listed 
Poor Placed / Poor Listed 
UI Claimants Placed / UI Claimants Listed . 
^ Migrants Placed^ /"fligrants Listed 

Women Placed / "^Women 'Listed 
Handicapped Placed / Handicapped Listed 
Youth Placed / Youth Listed 
^ Older'Workers Placed / Older Workers Listed 



Special Services 
9 



■ 10. 



■ Individuals Placed,: 

Applicants Tested + Counseled Enrolled in Training 

Applicants Tested / Counseled Enrolled in Training 
App^tic^nts Available ~ 



III. Internal Effectiveness Ratios 



IV. Quality 



11': New Applicants / Total Intake Cost (A) 

12. Referrals / Individuals Placed (B) 

13. Job Development Placements / Applicants Available (B) 

14. Inactivations / ES Positions (B) 

15. - Individuals Placed / Total Cost for Employment (A) 

16. In'dividuals Placed / Direct Employment Costs (A) 
17*. Openings Listed / Direct Employment Costs (A) - 
18. Individuals Placed / ES Position (B) 

1^ Long-Term Placements / Applicants Available (B) 

20. High-Wage Placements / Applicants Available (B) 



♦letters in parentheses indicate classificatibn of measures in the following criterion 
models: (A) Measures of output in relation to costs; (B) Level of performance measures; 
and (C) Penetration measures--extent of service. 
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very important. As.preVdous research has demonstrated, they have sub- 
stantial and highly significant relationships with the kinds of perfor- 
mance measures listed irtrthe table. 



Description of the Control Variables 




In the analysis of the control variables, previous research studies were 
reviewed to develop measures which potentially could have some impact on 
ES performance at the local office level. These measures included internal 
varia'Iples describing the nature of each local office, such as: 

Workload (Applicants Available/ES Positions) , - 

Claimants as a Percentage of Applicants Available ' ' 

Special Applicants (including Veterans, Minorities, Poor, 
UI Claimants, Migrants, etc., as listed in Table 9) as a 
Proportion of Total Applicants 

Since previous research had indicated that size of office was itself an 
important variable influencing performance, different size measures were 
also included in th| study, such as: 

• Applicants Available 

• ES Positions 

Also important in assessing ES performance has been the type of labor market' 
' situation in which the office is located. A number of variables were iden- 
tified and assessed to evaluate labor market factors. These included: 

• Total number of unemployed workers 

• Percent of workers employed in large firms 

• Percent of workers employed in manufacturing industries 

• Percent of workers employed in construction industries 
Unemployment rate • 

These control variables were analyzed in a number of ways to increase the 



understanding of their impact. The procedures used included the computa- 
tion of zero-order correlations between the individual control variables 
and the criterion measures; a factor analysis of the control scores ,to de- 
termine their underlying structure; and, finally, a multiple regression 
analysis of the control variables against two key performance measures. 

* *■ ■ 

Data Analysis of MAS Scores, 
^Control Scores, and Criterion Measures 

<^ 

The MAS results for the participating states were obtained by determining- 
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the percentage of favorable responses in each of the states on each of 

Uthe 19 score areas, so that some insight could be obtained about the range 
©f management procedures in the states studied, the; relationships between • 
the MAS scores and the control variables, and the validity of the MAS scores 
in predicting selected criterion measures were examined through correlations 
on the total sample. In addition, an analysis of selected validation results 
in selected states was carried out to clarify the implications of the research 

The methodological procedures followed in this study do not in any way convey 
the difficulty involved in working with these kinds of performance data. 
Initially, the investigators had been concerned that the' criterion data 
should be tightly sequenced around the administration of the Management 
Audit Survey so that the management procedures in effect* at the time could 
be related to the performance measures. This also obviously applied to 
the control variables. Thus; the, early data analyses carefully maintained 
the order of *jbhe criterion data in relation to the administration of the MAS, 
However, as the data analysis proceeded, it became clear that the error in 
the criterion data outweighed the time sequence of the criteria in relation 
to the MAS in importance. ^To illustrate, criterion data were obtained in 
Missouri. and Ohio for the year preceding the administration of the MAS. 
If it were necessary for criterion data to closely sWround the administra- 
tion of the MAS, the criteria from these states should, have been less pre- 
dictable than from states where the criterion data surrounded the MAS ad- 
ministration. However, it was found in the states where longer time periods 
were considered, e.g., the preceding year, ttiat higher validities w6re 
generally obtained. Thus, the concern in the literature about organiza- 
tional climate data being more valid for time periods considerably in the 
future or immediate in the past was nqt verified in this study. As a 
result of this experience, all of the quarterly ESARS data were collapsed. ^ 
The data were averaged across time periods when separate quarters were 
available. However, the quarterly data did allow analysis of the relia- 
bility of the criteria which was an important outcome of the study. 

Furthermore, examination of the data by quarters and by type of office 
indicated extensivv^ v^iation in the performance measures for local offices 
and for states."* the normalization procedure described below was very help- 
ful in removing some of the idiosyncracies in the d^ta. In addition, re- 
moving offices which had a specialized function, such as WIN, CEP, CETA, 
and other such classifications of offices, helped to make the remaining 
offices more comparable oi\< the'performance criteria. "Still, there were .7;'^ 
some extensive differences in offices by state. Either states used dif-, '"^ 
ferent kinds of data collection and reporting procedures, or there wei;^6* 
extensive problems in how the data were reported within the states. To 

"Resolve these, problems, the data were inspected by hand to remove highly 
improbable scM|es. This hand inspection only removed fbuiwoff ices from 
the total samj^e studied, but helped to make the resulting findings more 

^ representative. • 

Another procedure followed was the elimination of offices which had one 
or more impossible measures , i,e., placing more than 100% of thdir appli- 

etc.* The results of these screening procedures will be presented 
along with the discussion of the reliability of the criterion data. 



For a number of reasons ,_jiormalized scores were used in the analysis of 
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all data, lonless otherwise indicated. The shape of raw score distribu- 
tions, many of which were highly skewed in this ^udy, may have an undue 
influence on how variables are interrelated,. The normalization procedure 
rank ordered all of the scores, and then forced these rank orders into a 
normal distribution with a predetermined mean and standard deviation. 
This procedure controlled the effect of skewed distributions, the extremely 
deviant work group^, and made -the resulting vdistributions appropriate for 
a variety of statistical techniques. 

The correlations between the raw score and the normalized store for the 
criterion measures generally ranged above ,90, so the /procedure did not 
alter. the position of any one \foTk group in a rank ordering of all the 
work groups. These high interrelationships would normally indicate that 
either raw or normalized scores could be used, without any difference in 
the result, but the normalized scores were used for the reasons summarized 
below. ^ ; 



1) The normalized scores had a more varied pattern of interrelationships, 
suggesting higher levels of convergent and discriminanj: -validity and thus 
greater construct validity than did the raw score. ihtetX^l^-t ion ships . 

That is, conceptually similar normalized scores correl^t^cf rtor^ highly with 
each other tlfan did the corresponding r^' scores. * . ^ 

2) The normalizing procedure reduced, the effects oJf any LworR gTfoi^"S |:hat . . ^ 
were extremely different ffbm the other work gtbup^ in* a st^te.^ wwithput , . 
losing those work 'groupSi^?-^Iin^rcorrelation.^results*iOii^ meafeurfes 
reflect the .entire sampfe',' apid are not an*^ arj^ifacr of one or two extremely 

- dif iferent'work groups.. ^\ ^ .\ . . " 

3) Thfe^^normalized ;s,core^*mbre nearly meet^'^he :i^quireTnent^ of .statistical 
procedures (e.g. ^, hditioscedas^ici'ty) . v ^ ^ 

4) Normalizing ^^hi^ scores within states , allowed tBe.^data to be combined * 
aciiross stated' e^en, thougljr the criterion mjasures were for dSfferpnt tame 
periods, e;tc. '^^'^-'^ f'^ > ; ^ r-.X^J^' - 




CHAPTER V 



RESULTS , 

This presentation of the results will begin with the analysis of the cri- 
terion data. Initially' this will involve presentation of the reliabi- 
lities of the criterion measures and other problems encq^oitered with the 
data. The intercorrelations of the criterion measures will then be dis- 
cussed and certain other advantages and limitations of the various cri- 
terion measures will be pointed out. A discussion of control scores will 
follow and will involve zero-order correlations, a multiple regression 
analysfs of the control scores against selected criteria, and factor 
analysis of the control scores. Finally, the chapter will conclude with 
a presentation of the results cTbtained with the Manageme^nt Audit Survey. 
These data will be related to the control measures and to the criterion 
measures for the total sample and 'for selected measures by states. 



. \ ES Performance: Criterion and Control Scores 
Analysis of Criterion Reliabilities 

In four of the six states used in thi^ study— Mississippi , North Carolina, 
Tennessee, and Texas--criterion scores for different quarters were avail- 
able. Th4,t is, for the quarter preceding the administration of the MAS 
and the quarter succeeding the administration, separate sets of criterion 
data were available. On North Carolina and Tennessee, the second quarter 
preceding the administration of the MAS was also available. For, Missouri 
and Ohio, only yearly data were available. 

With repeated measures available^ over time, the reliability of the cri- 
terion measures could be determined. If the criterion measures were re- 
liable indicators of the -performance of ES local offices, then there should 
be a similarity between the scora reported for one quarter and the score 
reported for another quarter. Any differences found between the two quarters 
for a given work group might be accoimted for by error, "^^sonal differences, 
or by unusual circumstances, either in the local office orS^n the market 
environment served by the local office. With less^eliable criterion lileasures , 
predictions by any set of predictors ..re increa3ingiy more difficult because 
the scores are composed of error variance, not common or specific variance 
which would be stable across occasions. With unreliable data, the situa- 
tion is limilar to having a moving target, where the performances of the 
local offices within a state vary across time. 

The reliabilities of selected criterion scores for the four states are pre- 
sented in Table 10. These criteria were selected to represent different ES 
functions and to simplify the presentation of the results. These- selected 
criteria generally represented the more relevant criteria based on all the 
evidence available to the investigators which is presented throughoutV th^s^ 



Table 10 

Reliabilities of Averaged Adjacent Quarterly Data . . 
^ on Selected Criteria 



Criteria 



MS MO* NC OH* TN TX 

— ^ — - 



1. Direct Employment Costs 
per Placement 



.50 



.§0 



.32 .72 



2. Placements per Applicant 
Available 



.59 



.66 



,50 .80 



3. Openings Filled per 
C)penings Listed 



.58 



.77 



.69 .79 



4. Placements per Job 
Referral 



.59 



.75 



.50 .88 



5. Placements per ES 
Position 



»55 ^ .71 AU .74 



6. Proportion of Special 
* Applicants Placed 



.67 



.64 



.56 .86 



Number of Offices with Some 
Quarterly Data Dropped 



0 3 34 



12 ^ 



"^Yearly data only available, 
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chapter.' The selected criteria represented generally different kinds of 
measures for focusing on Cost. per Placement, Percentage of Applicants 
Placed, Openings Filled as a Percent of Openings Listed, two effectiveness , • 
measures of Placements per Job Referral and Placements per ES Position', 
and, finally, a special services measure, the Proplprtion of Special Appli- 
cants Placed. 

■ • 

The reliabiliti^es, where such measures were obtainable, varied by state. 
In Mississippi, all of the reliabilities for the selected measures were 
below :60,' except for criterion 6- -the Proportion of Special Applicants^ 
Placed-^which was ,67. While the reliabilities tend to be consistent, 
conventional standards for reliability are at least .60, if not highet. 
Thus, the reliabilities from Mississippi were low from a psychometric point 
of view. In North Carolina, quite a different picture emerged as all of 
the criterion measures had reliabilities greater than .60 and, in some 
cases, were as high as .80 (for the criterion measure of Direct Employ- 
-ment Cost per Placement). Tennessee was another state where reliabilities 
of criterion measures tended to be unustially low, ranging as low as -.13 
for Placements per ES Position. The Direct Employment Cost per Placemei^t 
also varied rather markedly, with a reliability coefficient of only .32. 
With low reliabilities, of course, the performances of the local offices 
approach th^t of random numbers and represent serious difficulties in the 
criterion measures. For Texas, generally satisfactory reliabilities were 
obtained as all o'f the reliability coefficients were above .70. This is 
a much more desirable state of affairs where effective criterion measures 
have been available. 7 

Since* quarterly data could not be obtained from Missouri and Ohio, relia- 
bility information was not available^ for these states. However, the yearly 
data would tend to be more reliable than quarterly data, averaging oufany 
unusual circumstances that might have occurred during that time. ^ 

Also shown in the table are the number of local offices v\ith some quarterly 
data dropped. These were local offices with performance measures which 
exceeded possible limits, indicating obvious errors in the reporting system. 
For example, such inaccuracies of measurement included more placements than 
applicants available, more openings filled than listed, mpre placements 
than referrals, etc. Although North Carolina tended to have generally reli- 
able criterion data, this was only after a number of offices had data deleted 
^Vhen final average data were computed for North Carolina, 34 offices had 
at least one. quarter of criterion data dropped because of apparent report- 
ing errors. Tennessee also had a fairly high number of errors in the data, 
as 12 offices had at least one quarter dropped. AltogethQ.r, 60 local offices 
had some quarterly data dropped because the reported scores were not possible 

In view of the relatively modest reliabilities and the number of errors en- 
countered in the data, studies of Employment Service performance are dif- 
ficult, due to the limited stability o-f the performaince measures. Also - 
difficult are policy decisions and higher level management procedures when 
accurate performance data are not available. These results suggest that: 

• Quality control procedures should be carefully reviewed so 

that more effective and accurate performance measures can be 
** 

obtained. ^ 
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• Alternate and simplified reporting procedures should be 
considered to obtain accurate performance measures of 

ES offices, both for use at the local and at the national 
level. 

• Important policy decisions should be supplemented with, 
additional information whenever possible to reduce the 
impact of any error inherent in the data. 

• A measure o^ the accuracy of the reported data should be 
SjHcluded as one of the measures of ES local office 

performance. 

In view of the limited reliabilities, the total sample of offices was 
further screened so as to exclude criTerion measures in selected states 
where the reliabilities/ were below .60. This 4)rocedure, in^effect, re- 
moved the majority of the measures from Mississippi and Tennessee but in- 
cluded essentially all of the measures on Missouri, North Carolina, Ohio, 
and Texas. The resulting sample size for the analysis of the criterion 
intercorrelations and fo^ the validity results presented later in this 
chapter was approximately 265, but some data were missing for some of the 
criteria. * 

Analysis of Criterion Interrelationships ' . 

The intercorrelations among the most relevant criteria (Variables 1 through 
6) together with correlations of, other criteria which had policy impli- . 
cations for ES (Variables 7 through 15) are presented in Table 11. Vari- 
able 1 (Direct Employment Cost per Placement) had a pattern df at least 
moderate negative intercorrelations with the other criterion measures in- 
cluded in the study. This measure of cost would be expected to have nega- 
tive intercorrelations as"- higher costs are a /measure of less effectiveness. 
The correlations, with Individuals Placed per Applicant Available, Percent 
of Opetiings Filled, and Individuals Placed per Referral were all modest. 
The correlation with Individuals Placed per ES Position, which is the 
major component of the cost figure, was considerably higher (-.66). 
Direct Employment Cost per Placement had a pattern of noticeably lower 
correlations with all of the rest of the criteria included in the study, 
indicating a lack of convei^gence with these criteria. 

Variable 2, Individuals Placed per Applicant Available and one of the key 
criterion measures in the study, correlated with Percent of Openings Filled 
at a modest level (.30) and higher with Individuals Placed per Referral 
and Individuals Placed per ES Ppsition (.56 and .51, respectively). These 
correlations indicated that there was some tendency among local offices 
that scored high on one performdnoe'' criterion to scorfe high on other quite 
different kinds of performance measures. However, the results also showed 
the complexity of E^^ operations', as a number of measures were necessary to 
adequately represent and assess ES performai^ee . Certainly these criteria 
represent an ^important spectrum of ES, activities. Individuals Placed per 
Applicant Available e<prrelated very markedly with the Percent of Special 
Applicants Placed, as might be expected, since they had similar components. 
The Percent of Applicants Placed also correlated highly with Variables 9 
and 10^ which also shared component^ with Individuals Placed per Applicant 
Available. , 

• « 
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Table 11 

Selected Criterion Score Intercorrelations 



Selected Criteria: 

1, Dir. Erap. Cost/Placements 

2. Indivo Placed/Apps. Avail. 

3. ' \ of Openings Filled 

4. Indiv. Placed/Referrals 

5, Indiv. Placed/ES Positions 

* 

6, \ of Spec. Apps. Placed 



I Less Relevant Criteria: 

01 

' 7, Job Dev. Placements/Apps. Avail. 



1 2 3 4 5 6 
f 



-25 " 

-30 30 " 

-39 56 64 

■66 51 34 45 -- 

-19* 94 30 50 44 



7 8 9 10 11 12 13 14 .15 / 



■09 34 10 22' 15 37 



8, 


Inact, with No Serv./ES Positions 


-12 -60 


13 


-12 


26 


-63 


-42 - 










9. 


Long-Term Placements/Apps, Avail, 


-12 82 


00 


28 


38 


80 


47 -65 










10. 


High-Wage Placements/Apps. Avail. 


-20 63 


-OS 

» 


16 


30 


61 


21 -43 


59 






i 


11. 


Apps, Avail. /Total Unemployed 


-12 42 


34 


38 


50 


32 , 


• -01 02 


14 


18 




t • 


12, 


Indiv, Placed/Total Employed 


-23 67 


50 


58 


65 


61 


-02 -23 


40 


40 


71 


WW 


13.^ 


Openings iisted/Total Employed 


-20 60 


34 


44 


43 


53 


-09 -37 


> 

30 


36 


64 


86.- 


14. 


Ind, Placed/Apps Tested+Couns+EnrTrng 


-09 39 


22 


23 


40 


'36 


22-09 


34 


15 


10 


21 11 - 


15. 


Apps Tested+Couns+EnrTrng/Apps Avail, 


03 04 


-12 


.04 


-16 


11 


02 -31 


12 


10 


00 


10 14-80 " 66 
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Variable 3, the Percent of Openings Filled, had modest correlations with 
the selected crfteria except for Individuals Placed per Referral, where 
the correlation was .64. This latter correlation is suggestive Of effec- 
tiveness in handling the characteristics of the employee and employer as 
the two measures of Percent of Openings Filled and efficiency in the place- 
ment process were substantially -related . 

The Individuals Placed per Referral measure (Variable 4) ^also tended to 
have gene'rinly modest correlations with the balance of the criteria studied. 
This process measure had some convergence, but certainly no duplication or 
substantial overlap with other criteria. 

Variable 5-- Individuals Placed' per ES Position--yas one of the most impor- 
tant criteria in thfe study, particularly because of its significance in 
the Balanced Placement Formula. This measure correlated highly with costs - 
and percent of applicants placed, and also correlated .65 with Variable 12 
(Individuals Placed/Total Employment). However, the latter correlation is 
partially due to shared terms, the numerator, which are very^ important in 
interpreting these data. 

The composite Percent of Special Applicants Plaiced (based on the number of 
Minorities, Youth, etc. placed, divided by the .(^rresponding; .number listed) 
was so highly related to Individuals Placed per Applicant Available (Variabl( 
2) that it could have been elimingrfeT^om the study. However, it does 
represent an important function o^ ES services and thus was retained. The 
measure was most related' to other measures which contained similar terms, 
having placements in th'e\ numerator and the denominator consisting of 
applicants available. / ^ 

Criteria 7 through 15 w^ac4 measures which were examined throughout the 
course of the study, but which seemed to have somewhat less relevance for 
ES operations as standards of effective performance. Some of these measures 
do, however, have important implication^ for ES >0perations. For example. 
Variable 8, the number of individuals Inactivated with no Service per ES 
Position, had a pattern of somewhat marked correlations with other criteria, 
thus suggesting its possible use as a measure of ineffectiveness. However, 
close inspection of the relationships indicated that a number of the cor- 
relations w^re relatively low (e.g., with Percent of Openings Filled, the 
corr lotion was only .13). In addition, this measure was substantially re- 
latec tV.^ control Variable, Workload (r = .91). Because this criterion 
was s highly contaminated by Workload, measure was not furth&r consi- . 
dered as an important criterio^ measure. , 

The mwo quality measures (Variables 9 and 10),- dealing with the number of 
long-term and high-wage placements per applicant available, tended to have 
generally modest relationships with other variabK^s, except for the marked' 
correlations with the other percentage of applicants placed measures. In. 
addition, these quality measures had relatively .low means. Other analyses, 
not presented here, examined high-wage and long-term placements as a percent 
.of total placements. The results showed marked variations between local 
offices, and these measures had zero or negative, relations to performance 
on the other selected criteria. These Ifesults^^^gest that high-wage and 
long-term placements are not an important integrated part of ES operations , 

67 
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:inO warrant more attention both in future' research and in the practices 
of management to brinj^ about higher levels of performance on these quality 
of peri-ormance fnectsures, ' 

Variables 11, 12, and 15 were all penetration measures dealing with the 
amount of ' service rendered by the local office in relation to the amount 
of economic activity in the county served by the local office. These 
measures had some problems associated with them, even though the correla- 
tions with the .selected criteria were generally satisfactory. . When the 
local office is the unit of analysis, matching the local or^ic^ 'in an 
appropriate manner with the county served is difficult. Thus, at the 
local office iQved, there were only a limited number of offices (67) which 
could be accurS:teiy linked to a specific county where economic data were » 
ayailable. In ..all other cases, there were two or more of fices ^in. one 
covinty or one p_%fice serving a number of counties and the lines of service 
could not be clearly identified. In handling these data, only offices 
serving one county were considered. Thus, the « sample for these measures 
was small. These measures were relatively strongly influenced by office. ' 
size^, correlating generally In the -:50's with the denominator, indicating 
that the offices in the larger cities were less efficient on these ^criteria. 

The last two variables in the matrix, 14 and 15, were concerned *with spe- 
cial services given to applicants. Variable '14 dealt with the number of 
placements expressed as a percentage of the. number of applicants tested, 
counseled, or enrolled in -training. Variable 15., the number of applicants 
given these special services expressed as a percent, of applicants available, 
was-av..simiiar measure.- The latter measure, dealing with the extent of service 
to applicants, had, zero or very low correlations with all the other criteria 
except Variable 14 with which it shared common terms. Although Variable 14 
looked somewhat more relevant, with somewhat higher intercorrelations, the 
placement term in'the numerator was a contributing factor in 'the results obr* 
tained. In view of the essentially zero relationships of Variable 15 with , 
all other criteria, the results indicated that the expenditure of ef^fort 
on testing, counseling, and t^'aining of applicants did not tj^rtd to result 
in higher levels of HS performances. These services might b6~ justified 
on other grounds (e.g., better service to applicants), but there was no 
immediate observable payoff in ES performances. ' : . 



These results suggest that: 



ES performance consists of a numbed of complexly inter- 
related components. The und"erlying management practices 
which can have an effect on one aspect of performance ^must 
be considered for their pote'ntial effect on 'other important 
performance areas . ^ . ' " 

Measures of cost of ES performance are related to delivery 
of sevfice, suggesting that those management practices 'wto:6h 
underly ,cost effectiveness are either related to other 
practices affecting non-cost measures of performance, or . 
have a general effect across areas of performance. i 

The quality of placement measures (high-wage and long-term 
placements) haid limited relationships to the other ES per- 
formance criteria, i.e., high levels of performance on these 
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, measures did not go hand in hand with high levels of 
other performance indicators. Management practices 
which emphasize and increase this area of perfor- 
mance should be considered if high quality of place- * 
ments is to remain a goal of ES offices. 

• The" modest but consistent relationships of the efficiency 
measure of Individuals Placed per Referral with the other 
^. " criteria indicated that 'general efficiency o£ operations 
was an important contributor to the performance of ES 
offices across • measures . 

Criterion-Control Score Relationships 

The intercorrelations^of selected control scoi/es and their relationships 
with^^5ele^^4 crit^ggii^ are presented in Table I2-. During the course of 
the study, a TaT'^ge number of control scores were examined^ for their in- 
fluence on performance measures of £S offices. T^ically, these measures 
had an interesting pattern of relationships with "selected criteria. 

/Workload, which was assessed by the ratio of applicants available to ES ' 
positions, had a very meaningful pattern of relationships with the selected 
ci;iteria. To illustrate, high Workload, where there wejre^ a;, great number of 
appl^icants to serve in relation to the staff available, resulted in lower 
cost per placement, i.e.,<|:here was a positive Ijenefit in having high Work- 
load. But this benefit was associated with a lower percent of applicants 
placed (r = - .46), The correlations of Workload with variables 3 and 4, 
Percent of Openings Filled and Individuals Placed per Referral, were essen- 
tially zero, making no difference. But on variable 5, Individuals Placed 
per ES Position, a positive relationship was again obtained = .44]. 
This result indicated that high Workload was associated with another form, 
of effectiveness, a productivity measure of Individuals Placed per ES 
Position. Workload was also correlated -.50 with the Percent of Special 
Applicants Placed. Because Workload had a differential impact on two im- 
portant goals of ES operations, this phenomenon will be discussed in 
greater detail later. . * - 

Variable 8--Ui Claimants per Applicant Available--also had significant 
' correlations with some of the selected criteria, correlating most highly 
with Variables 2 and 6, which dealt with the percentage of applicants 
placed. ^1^^^ correlations tended to be very low, indicating an inconse- 
quential impagt. The results obtained with percent of UI claimants and 
percent of applicants placed indicated that local offices with a large pro-, 
portion of UI claimants in the total applicant pool placed a lower pe:^entage 
of their applicants. For reasons which cannot be determined from the 'data," 
these individuals were apparently more difficult to place. 

Variable 9--Proportion Special Applicants--had a pattern of relationships 
with the selected criteria similar to, but somewhat lower than, that of 
Ul Claimants per Applicant ^Available. The exception to the pattern was with 
. Individuals Placed per ES Position; special applicants were apparently 
somewhat more demanding of ES staff time resultinfl^dn lower productivity 
(r = -.33). Variables'lO and ll--the two size m^^Ures of Applicants Avaii:^ 
-able and ES Positions--had very similar patterns of^^ correlations with the 
selected criteria as would be expected since the correlation between these 



-59- 



Table 12 



Intercorrelations of Control Scores 
with Selected Criteria 

r 





Variables ■ . 


7 


8 


9 


10 


11 


12 


• 13 


14 


" 15 


16 


Criteria: " 






















1. 


Direct Employ. Cost/Placements 


-37 


16 


19 


32 


47 


35 


17 


. 01 


-02 


-0^ 


2. 


Indivi Placed/Apps, Avail, 


-46 


-52 


-40 


-35 


-10 


-48 


-16 


03 


00 


-34 


< 
3. 


% of Openings Filled " / 


07 


-08 


04 


-2a 


-22 


-50 


-21 


05 


-10 


. 03 


4. 


Indiv. ;Placecl/Referrals 


-12 


-21 


-08 


-38 


-2^ 


s;56 


-29, 


J32 


05 


-03 


5. 


Indiv. Placed/ES Pj^sitions 


.44 


-18 


-33 


-18 


-39 


-34 


-22^ 


f03' 


03 


-07 


6. 


% of Special Applicants Placed 


-50^ 


MS 


-38 


-24 


02 


-35 


-06 


109 


07 


-31 



Control Scores : 

■ a ' 

7. Workload (AA/ES. Positi6ns) 



8. 


UI Claimant/ Applicants Avail, 


37 












9. 


Special' Apps/Appli cants Avail. •. 


-10 


22 










10. 


Total Applicants Available 


.i2 


08 


20 








11. 


Total ES Positions 


-36 


-07 


14 


, 82 






12. 


Total Unemployed 


06 


22 


1.3 


. 6,6 


53 




13. 


% Employed in Large Firms 


-08 


00 


-08 


' 26 


27* 


46 — 


14. 


% Employed in Manufacturing 


-01 


13 


-08 


-04 


05 


bij^zi -- 


15. 


0 Employed in Cons±ruQtion 


-06 


-07 




28 


31 


25 12 -08 


16. 


UnemployTiien||t|B±e ) 


35 


31 


30 


09 


-02 


23 -08 18 
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•two size measures was .82. CertainlWT^he general indication of the re- 
sults for these two variables was tlyit large offices were less efficient 
on the criteria studied. In g.enej^, both of these size measures had neg- 
ative relationships with each of the selected criteria. 

Variables- 12 through 16 all dealt with characteristics of the community in 
which the local office was located. Variable. 12 assessed the total number 
of unemployed workers, within the community and tended to -have a consistent 
and fairly strong negative impact on ES performance. All correlations 
were above .30 in magnitude, indicating that, in counties where there were 
a large number of unemployed workers, the local offices tended to perform . 
less weir on the performance criteria. ,This . result , to some extent^ paral- 
leled those from the other size measures . Variable 13, the percent employed 
in large firms' within the county served by the local office, might h^ve * 
been expected to have positive relationships with ES performance, since 
it would indicate a limited number of firms to be contacted and to serve. 
,Yet, contrary results were ol5tained, evidently because the large firms 
tend. to be located in areas where there is also a large number of unemployed^ 
workers, i.e., the correlation between the Percent^tmployed in Large Firms ^ 
and the total number ^of unemployed workers withinithe county was /46. 
Although the percentage of workers employed in la/rge firms had a. negative 
impact, the correlations did tend to be modest. Variables, 14 , 15, and 16 
all generally had rather slight impact on ES performance, although unemploy- 
ment rate did Correlate -negatively with the percentage of applicants placed 
and the percentage of special applicants placed. 

These zero-order relationships provide the cleaffest interpretation of the 
impact of the control variables considered singly. However, they do not 
provide information about h^w\these variables interact to produce a cumula- 
tive effect on ES performance .\ This topic will be treated later in the 
report. . \ . 

As mentioned previously. Workload correlated -.46 with Percent of Applicants 
Placed and .44 with Individuals Placed per ES Position on the combined 
states sample. These results replicated almost exactly those found in the 
previous study on Pennsylvania, where the corresponding correlations were 

.44 and .^i:.'' In view 6f this differential impact of Workload on two impor- 
tant pcrfuimance measures of ES operations, the topic will be treated more 
extensively. 

In Yal)}e i7>y the regression lines for two criterion measures using the 
noriiiaiized scores have been plotted on Workload- using raw scores. The data 
displayed in this table were taken from Ohio and represent the general find- 
ings which occurred for the total sample of offices studied. The table shows 
what level of performance on each of the two criterion measures could* be 
expected from various levels of -Workload. At low levels of Workload (ap- 
proximately 600 applicants per ES position on yearly data), an above average 
percent of applicants were placed, but a below average rate of productivity 
was obtained. At the other extreme, where ther^was an extrei^ly high Work- 
load, the opposite phenomenon occurred. Of f ices a^^ear very efficient on the 
productivity measure of Individuals Placed per ES Position but are markedly 
below average in placing applicants. Thus, the level o^ Workload should 
be adapted, depending on national priorities, goals, and the relative im- 
portance of these two different kinds of criteria of ES performance . If 



Table 13 



Ohio Regression Results and Correlations of 
Workload with Two Key Criteria (Yearly Data) 



160 




r = .39 

Individuals Plac< 
Pqr ES Position 



r = - . 37 
Pla'cements Per 
Applicant Availal 



600 



1100 



i6a(r-. 



2100 



2600 



3100 



WORKLOAD (AA/ESP) 



ercent of Units 16°a 



33% 



32% 



08% 
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these goals were judged to be equally important, the approximate staffing 
pattern of Workload should be near the mean of the Workload variable, where 
the two regression lines would be expected to intersect. Since common data 
by. state were not obtained on the number of ES positions, the current study 
cannot be used to set or 'even approximate the optimum level of Workload, 
given a definition of the goal importance.. However, an important topic for 
future research would be a national study of Workload to examine the impact 
on these two criteria. In this manner, guidelines might be established for 
states and local offices to develop more nearly optimum leve'ls of Workload, 
which are congruent with national policy. In addition, since Workload could 
be expected to vary markedly across seasons, depending on the' local office, 
these results would also strongly encourage the use of flexible staffing . 
procedures to minimize costs while still maintaining effective performance. 
If flexibility of^affing can be obtained, it should represent a signifi- 
cant benefit to EB performance. 

/ 

The analyses of the criterion-control score relationships led to the follow- 
ing conclusions: ^ , - 

• Workload is a very important variable in determining the 
effectiveness of ES offices. High Workload has a , negative 
impact on Individuals Placed^ per Applicant Available and a 
positive impact on Individuals Placed per ES Position. Thus, 
the level of Workload should reflect national priorities. A 
consequence of this result is that states, to maintain a high 
. level of funding, could withhold or add staff expenditures ac- 
cording to the importance of the two goals in the allocation 

f funds/ Currently, Individuals Placed per ES Position has ^ , 
a high weight; thus, a high Workload would tend to result in 
higher funds and a lower percent of individuals placed. 
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• /A higher proportion of special applicants--vomen , handicapped, 

youth, older workers, minorities, veteran^, poor , and UI' 
claimants-- in the applicant population placets greater demands 
. on the ES staff resulting in lessened productivity.. Larger 
;JS offices generally tend to have more special- applicants 
and may* thus be penalized for their lessened efficiency 
with decreased funds. 

• T}\e proporti(^i^\of UI claimants in the applicant population 
' served also leads to lessened productivity on the basis of 

percentage^ of applicants placed. Further research in this 
*area is/warranted. 

• The characteristics of the types of employment in an ES 
office's area had only a slight influence on productivity. 
Size of the office was more important, with a moderate 
negative influence on ES performance measures. 

• The control scores investigated in this\jHiSy. provided an 
explanation for a large part (approximately\56&) of the 
differences in the performance of ES offices^on two 
important criteria. Control scores should be included in 
any investigation of the effectiveness of the E9- . How 
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• * • th*esi5.^|ont;(vo.l' score effects can be oyerbgfae; t)y 'ES. 




' * '-^fe^''^^ admiivf Orators) to iinprbye performance shou^^ 
/j^^ . topic fot further research. 





/^Multiple Regi|$ssicm)of C6nti4)l ScofeSj-jn^ 

Ten coniroi scfore^s were placed^ iii^«"'stepwis^,r^^ r€f;^ess^6: 
t(\ predict eacly of '^wo selectjf'^'/pi^o^^ Indi'vi^al 
Placecf^per Applic^ Avairab^e;i^^d-^23. in:dividuais Pta^^d^hr ES^'fositu 

. The results of the|e analyses aS^ pr^esen t£^b,les. 14 and 15. " %es^ 

. tobies shq«r tihe 2:ei:o Vljd^'W 

beta r'^produ6ts,r:'and -the"' tavv s(cora weigh ts^Sljen ail 1 'Variables had beefi 

normdlized with .a ineau of IffO and. a standard deyiat^n of. 2l3* , - 

„ ' »- > \ .. 

For the Indivij^als Placed' per>Applicaht Available .crii:'ejrion,. the multiple 



correlation was .72, aCcountitig for 52% of the ^rlance.' 'Claimants per 
Applicant Available and Workload had considerably tiighef beta r j)roducts 
than the other control scores, together accounting for approximately 30% 
of 'the^criterion variance. The variables entering into the equation had 
. * negative weights, with the exception of Percent 'Employed in Manufacturing 
and Percent Employed in Construction, which" were positively -coj^re^ated 
with the criterion. The absolute contribution of these latter two variables 
to the prediction of the criterion was small. Having eight variables^ entered 
into a multiple regression equation with significant weights is relatively 
rare. These results indicate the complexity of the criterion measure, with 
a large number of variables having significant effects. 

For the other selected criterion of Individuals Placed per ES Position ,( 
the multiple correlation was .68, accounting for 46% of the variance. 
Again^ the first two variables entered into the prediction equation were 
Workload and UI Claimants per ^plicant Available. Here, however, their ^ 
order Was reversed, and Workload had a positive, rather than ^egative, im- 
pact . The relative contribution^ of Workload to the total prediction was 
; ' substantial when compared to the other predictors. The beta r- product for 

Workload was .26, while th'e next highest contribution for the other variables 
was .06. These data confirmed the findings presented above concerning Work- 
load and its impact as an important control score on performance indicators, 
oversh'adowing all the otjier control scores. 

V/hile the relationship of the best weighted combination of control scores 
to the criterion measures is substantial , this area^ of investigation should 
be studied more- closely and these results interpreted cautiously. For example 
Workload correlated -.92 with Direct Employment Cost per Placement in North 
Carolina, but only -.04 in Ohio. If the criterion measures were relatively^ 
stable across time and across geographical locations, thie magnitudes of 
the criterion- control score correlations should be much more comparable. 
The relationship between Individuals Placed per job Referral, another cri- 
terion measure, and the' control score^of UI Claimants per Applicant Avail- 
able provides additional evidence of lack of stability in fh^criterion- 
control relationships; the correlation was -.49 in Missouri ,iH4t_only -.07 
in Ohio. ^ 

So, while control scores are important and must ^e taken into consideration 
in any investigation of, the effectiveness of ES performance, the impact of 
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Table 14 



'Multiple Regression Results: The Prediction of 
Individuals Placed per Applicant Available by the Control Variables 





Predictors ^ 


Validity ' 


- 

Beta , 


Beta r 
Product . 


Raw Score^ 
Weight 


4. 


UI Claimant per Applicant Avail. 




-.3490 

■,: -0 '. 


.1814 


-.328 


T 

J) . 


nUlIxlUclLi ^/vr\/ cor ) 


- .45 


t 

-.2571 


.1157 


- . 204 


5. 


Proportion Special Applicants 


-.38 


-..1968 


.0748 


. -.163 


6. 


Total Applicants Available 


-.32 


-.2177 


.0697 


-.173 


10. 


% Employed irl Manufacturing 


.11 


.1958 


.0215 


.157 f 


7. 


Unempl^^ent Rate 


-.36 


• -.1322 


.0475 


- .109, 


8. 


% Large Firms of Total Firms 


-.03 


-.1308 , 


.0039 


-.095 


11. 


% Employed in Construction 


.04 


' ;1014 


.0040 


.093 










.52 


/ 




R 


1 




.72 





— . ^ : 

Note: Both tlie criterion and the predictors were normalized and given a 
mean of 100 and a standard deviation of 20. The contribution of 
each variable tb the regression was significant at the .05 level. 
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Table 15 

Multiple Regression Results: The Prediction of 
Individuals Placed per ES Position by the Control Variables 



Predictors 



Validity 



Beta 



Beta r 
Product 



Raw Score 
Weight 



3. ^Workload (AA/ESP) 

4. -UI Claimant Per App. Avail. 
•5. Proportion Special Applicants 

10. % Employed in Manufacturing . - 
.9. % Employed in Large Firms • 

- 7. Unemployment, Rate 

11. h Fimployed in Construction 
6.^Total Applicants Available 

■ ' r2'^ ■ • o ■ 

R 



.42 


.6135 


.2577 


.616 


-.20 


-.3263 


. .0653 


- . 388 


-.29 


. -.2042 


.0592 


-.214 


.11 


.2286 


.0251 


.232 


;- .1,2 


-.1533 


.0184 


■^.120 


-.08 


-.1591 


.0127 


-.166 


.07 


.1689 


.0118 


.196 


-.14 


-.1283 


.0180 


-.129 






.46 








.68 





Note: Both the criterion and the predictors were normalized and given a 
mean of 100 and a standard deviation of 20. The contribution of 
each variable to the regression was significant at the ,05 level. 



er|c/ 



these variables in a particular situation could be expected to vary. 

These findings should be interpreted in conjunct ion ^with findings on re- 
liability of the criteria, i.e., approximately 3A to 40% of variance was 
unstable, so that the remaining variance not subject .to error or control 
scor^^ effects is small. Stated alternately, that portion of the total 
variance subject to management control and prediction by the MAS is small, 



These findings indicate that:. 

• Control Scores had an importantJ|fcw:t on ES performance 
but the -criterion-control syst^MHjT relatively unstable, 
subject to a substantial amount^r^error or variability in 
a particular state or in a given time period. 



• Predictions of these relatively unstable criterion measures 
with the MAS' scores when control scores had such a varied 
effect were relatively difficult. 

^ These results, based on local offices, as the unit of 
analysis, may be| more or less stable when the data are 
aggregated to higher levels of analysis such as SMSA's 
or to states. Certainly this is a^Kf^mising area for 
future research.^ 

• The importance of developing a simplified and accurate 
criterion reporting system that is responsive to the 
needs of ail level-s of m*anagement is reinforced. Such 

a system could be used to integrate organizational goals, 
management procedures, and ES performance. ^' 

Fact or A nalysis of Control Scores > 
ft ■ • 

Selected control scores in the validation study were factor analy^zed to 
examine the underlyirlg dimensions and common characteristics among the 
various measures. Table 16 presents the rotated factor matrix for these 
control scores and the communality for each variable. The communality is 
an estimate of the reliability of that variable based on shared common 
variance and is typically an underestimate of other forms^-of reliability. 
Only those loadings above .40 were included in the matrix to simplify pre- 
sentation. 



Factor A was a size dimejj:^^f!^|Rre^mined by the potential number of 
employable people who couiy use the>. services o£ a local Job Service office, 
as w61I as the size of the local oft^Nce. 

Factor B was somewhat complex and dealt .wi4^i_t he nature of employment in 
urban-rural locations . High percentages of tTTe-4abor force being employed 
in governmental -work and in manufacturing had opposite signs, indicating 
that an area with a large number of workers in government tended to have a 
smj/ii- number of workers in manufacturing. 

Factor C was a workload dimension and none of the other control scores had 
loadings greater than .40 on it. These results indicated that office 
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I Table 16 

Factor Analysis of Control Scores-- 
Rotated Factor Matrix and Communal it ies* 





• 


^ A 


B 


C D 


• 


7. 


Total Employed 


.94 






.91 


15. 


Civilian Work Force 


.'94 






.91 


6. 


Total Unemployed 


.91 






.90 


5. 


Applicants Available . 


.81 






.69 


11. 


Percent Employed- Large Firms 


.48 


.67 




•72. 


15. 


P.ercent Employed-Construction 


.43 






.26 


9. 


Change in Emplayment 


.42 




.44 


.42 


10. 


Percent Large Firms of Total Firms 


.40 


.69 




.6,8 


12. 


Percent Employed-Manuf acturing 




. 77"^ 






14. ^ 


Percent Employed in Government 




-.77 




.62 


1.- 


Workload (AA/NIAS N) 

» 






.91 


.85 


2 . 


Workload (AA/ES Position) 






.89 


.84 


.3. 


AA-UI Claimants per AA Total 






.70 


- .57 


8.. . 


Unemplo)'Tnent Rate 






.67 


.56 


4. 


Proportion Special Applicants 






-.65 


.49 



^Factor loadings below .40 omitted. 
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procedures with regard to. Workload were not, strongly associated with th^ 
location or size of office. 

Factor d' was a special applicants dimension. Special applicants Include 
those who', were female, economically disadvantaged, minority group members, 
handicapped, veterans, under 22, over 45, and/or UI claimants. The factor 
was also defined by the unemployment rate and change in employment. The 
factor thus represented a dimension dealing with different applicant groups 
^rved. n 

Results of the MAS 

Tills section will proceed by presenting the results of the percent favorable 
scores across the six states studied, where wide differences were obtained. 
This will be followed by a relatively brief section on thtf relationships 
of M'\S to the control scores and conclude with a rather extended section 
on the validities of the MAS in predicting the criterion measures, both 
on the combined sample and in .selected states for selected criterion 
measures . 

Percent Favorable Responses .... 

Tlu- six states in this sample had wfde differences on the percent favorable 

scoi'-es as shown in Table 17. Mississippi had the highest score on 11 of the 

19 MAS score areas, while Oliio had the lo^^st scores on l.-^ MAS measures. 

Fven thoLigii Mississippi was the "highest overall on percent favorable, 

that staL'o was also the lowest on EEO for Women and EF.O for Min- 

orities so th'ere is still room for improvement. Ohio had the highest 

score on hEO for Minorities, and some other scores were above or near average. . 

There have l,ee7recent ch^iges in the top administration in the state _ 
[;,npioymenf Service in Ohio, so tl^^se low scores may have ^^anged since the 
MAS administration. In a managem\it and administrative review of the Bureau 
of Hmplovment Services In the State of Ohio , conducted by the Region V Manpower 
Vlndnistration ( 1975), 52 separ^ite recommendations were made in an ettort 
to increase the ef ! uctiveness of that Bureau. Over l/-Vof tliose reco>mnen- 
dations would have a direct and immediate impact on the management practices ^ 
measured by ti:e MAS. 



For example, the first recommendation made in this teport stated that the 
role and re^poas.bility of the administrator's office s}iould be clear yde- , 
fined because the review found that the administrator's office tended to 
manage day-to-dav operations, even in lo.cal offices, undercutting Division 
.Sfrectors' Di str Lct'oirecW s , and loc al office managers. The ^^plementa- 
tion of this recommendation would hav^-arr-obvi ous effect on the NIAS score 
of Delegation of Authority, with some overlap on the scores f 
and .^dmtalstrative Ffficiency, Supervisory Effectiveness and Morale F rther 
i-esearch should determine if the recommendations were f° 1°^^, and if this 
did have an otfort on the management practices measured by the MAS. 

On the individual MAS score areas, the highest percent favorable scores 
were found on Wor!< Satisfaction, Operational Efficiency, and Lo-Worl<eT 
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Table 17 





Percent of Favorable 


Responses 


for 


the 19 


MAS 


Scores 






ft 


. Across 


Participating States 














' MS 


MO 


NC 


OH 


TN 


TX 


AVERAGE 


MAS 


Score Areas: ^ 
















/ 


Fairness of Management. ■[ 


DO 


DO 


Dy 


58 


65 


65 - 






Delegation of Authority 


D / 


c c: 


D / 


49 


59 


55 


* 55 


i 


Supervisory Effectiveness 


A- 7 
D / 


DO 


DU 


60 


66 


66 


64 


4. 


Planning S Admin. Efficiency 


A 7 

47 


A Q 

4o 


4 o 


42 


48 


46 


46 


5. 


•Climate' for Innovation 




/I y1 


40 


36 


47 


43 


44 ■ 


6.- 


WcJrk Satisfaction 


75 


75 


7 A 

/4 


72 


76 


71 


74 


7. 


Training Effectiveness , 


bo 


4 5U 


O Z 


"^5 


52 


51 


51 


8. 


Performance Feedback 


48 


4^ 


4 0 


"37 


' 48 


44 


44 


9. 


EEO for Women 




A- 7 
DO 


7 1 


66 


73 


64 


65 


10. 


EEO for Minorities 


61 


7 1 


AQ 


72 


71 


70 


69 


11. • 


Opportunity for^^-Prbmotion 


33 


28 


31 


20 


29 


32 


29 


12. 


Downward Communication 


67 


61 


62 ' 


59 


66 . 


61 


63 


13. 


Upward Communication 


52 


46 


46 


40 


51 


46 


47 


14. 


Satisfaction with Pay 


32 


26 


26 


9A 
^D 


7Q 


44 




15. 


Morale 


1,- ^ 62 


57 


50 


43 


58 


56 


54 


16. 


Physical Work. Cond. S Equipment \ 52 


59 


. 46 


45 


45 


55 


50 


17. 


Co- Worker Cooperation 


..5\74 




70 


64 


70 


.72 


70 


18. 


Operational Efficiency 


■ 75 


73 


68 


65 


72 


71 


71 


19. 


Workload Balance 


55 


52 


40 


41 


45 


43 


""~^6 




«» 

"d Understaf fing Responses 


37 


40 


55 


51 


50 


50 






■0 Overstaffing Responses 


8 


9 


5 


'V 8 


5 


7 

'4 
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Cooperation. These scores would indicate that the ES employees in these 
six states find satisfaction in the work they are doing, feel that their 
units function veryjwell, and cooperate with e^ch other in th^r work. 

^ I:ow scores were found on Opportunity for Promotion and Satisfaction with 
Pay. These two scores are generally not controllable by the unitr super- 
visor, so action strategies that could be used to counteract these low 
scores are difficult. The state-level administrators should be aware of 
them, however,, and either move to alleviate these problems, or provide 
more explanation for the employees so that they can understand why promo- 
tions and salaries are not as frequent nor as high as the employees would 
like. f * ' 

Among other scores that are low are Performance Feedback, Climate for 
Innovation, Planning and Administrative Efficiency, and Upward Communication. 
These scores could be improved with different management practices, and the 
MAS can used to suggest management action strategies that could be used 
to result-- in an improvement in these score areas. 



The Rej^ionships of MAS Scores to Control Scores 

The relationships -among the MAS scores and the control scores provide some " 
explanation concerning the effect of environmental variables on management 
practices. Certain environmental variables affect MAS scores in different 
ways, and knowledge of these control scores would help a supervisor to 
evaluate Ids managelnent practices, as measured by the MAS, and to determine 
what the sup^ruisor can do to improve the performance of the ujiit . 

For example, 1 argcr of f ices , as defined by Applicants AvailaJ^le and number 
of ES Positions, had low MAS scores on Fairness of Management, Delegation 
of Authority, Performance Feedback, Downward Communication, Upward Communi- 
cation, and Morale. -These results do not indicate that a supervisor of a 
large office can not develop action plans that will improve the scores for 
thfese management areas. On tHe contrary, problem areas can be identified 
and steps taken to improve performance in these areas. 

In the earlier study in Pennsylvania, Total Civilian Work Force had a strong 
n-egative reiatUonship t Satisfaction with Pay. This finding was not repli- 
cated in this study on the total sample, but the- results in Ohio were similar 
to those obtained in [Pennsylvania. Total Civilian Work Force did -have signi- 
ficant negative relationships with eight MAS scores, including Performance 
Feedback, Opportunity for Promotion, .Downward Communication, Upward Communi- 
cation, and Morale. The larger offices located in large urban centers have 
more urgent problems than ot^ier kinds of offices, and more time should be 



devoted to tft^^ management areas. 

High Workload (Applicants Available per ES Position) was related to low 
Workload Balance--an indication of the conceptual validity of the MAS, and 
. of the reliance that can be placed on the perceptions of the ES employees. 
On tlie total sample, the Workload cont^rql scare did not correlate signifi- 
cantly with an/ of tlie other MAS scores, a finding that did not replicate 
the earlier results in Pennsylvania . 

Total Unemployed had negative correlations with 11 of the 19 MAS scores as 
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.would be expected in view of the correlation between Total Unemployed and 
Applicants Available (r = .66). ^ 

Two control scores were found to account for more variance than other con- . 
trol scores in the multiple regression analyses to predict Individuals ; 
Placed per Applicant Available and Individuals Placed per ES Position • ..l-.V 
These we^e, Workload and UI Claimants per Applicant Available. These 
measur^Ijf^d, very low relationships to theVMAS scores except for •the Workload 
score ilC't(he MAS. This indicates that the YlAS scores ind the control variable 
were measipring different parts of the variance of the criterion measures--at 
least th^itw^?"' for which the multiple regression was run. 

In generafv trher findings of this study f>artially corroborated the findings 
of the earl^ter Pennsylvania study, especially concerning the effect of office 
size. The accumulation of more reliable data in the future will be needed 
to find more answers concerning the effects of situational variables on 
management performance. Conceptually, the findings o? this study make l9gical 
sense, and 'provide a more adequate explanation pf the effect ot the control 
variables on the measures of management practices. 

* . ^ 

Validity Results: MAS, Versus Performance Criteria 

The validity coefficients of the 19 MAS scores against the six selected, 
performance criteria are presented in/Table 18. In th,is table of 114 validity 
coefficients, approximately 5% of them could'be expected to be significant 
by chance. The results indicate a generally high percentage of significant 
validity coefficients, but the correlations are of limited magnitude. With 
the exception of column 1, which represents a cost .criterion with which the 
correlations were expected to be'negative, all other correlations in.the 
table reflecting positive attributes of ES performance shoul'd be positive 
and this is generally true of the results that were obtain^ed. 

The most valid score for the Direct Employment Cost per Placement criteripn*' 
was Performance Feedback, which confirmed the general findings of the Penn- 
sylvania study. Altogether, 32% of the MAS scores had significant correla- 
tions with this criterion- Other MAS scores which correlated significantly 
with this criterion measure included Fairness bf Mana^gement, Delegation of 
Authority, Opportunity for Promotion, Upward Communication, end Operational 
Efficiency. * 

With the second criterion, the number of Individual ' Placed per*" Applicant 
Available, 42% of the MAS scores had significant re ^ationshii;^, the two 
highest correlations were with Satisfaction with Pa> and Of^erational Ef- 
ficiency, both of which correlated at the .22 l^vel. 

For the -third criterion. Percent of Openings Filled, 74% of the validity 
coefficients were significant, with Morale and Opportunity for Promotion 
being, the most valid scores. Individuals Placed per Referral was this most 
predictable critexi-on in the study on the total sample, as 79% of the 
MAS scores had significant validity coefficients against this criterion.^ 
Thc'.niost valid score was Motile followed closely by Operational Efficienfcy 
and a number of other, MAS scores. Individuals Placed pe?- ES Position, one. 
of the more important criterion scores in the study, was predicted with ' 
significant validity coefficients ^by 58% of the MAS scores. The mSst vali^fl^ 
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Table 18 

Validity Coefficients of MAS Scores 
/in Predicting Selected Criteria 

(Combined States Data; N between 170. and 245) 



7^ 



Fairness of Management 
Delegation of Authority 
"Supervisory Effectiveness 
in. § Admin. Efficiency^ * 
f sX Clirhate for Innovation 
Worlt Wtisfaction 

7. Training Effectiveness 

8. Performance Feedback 
^9. EEO for Women 

UO.^ "Geo for Minorities 
11: Opportunity for Prjpmotion 

12. Downward Communication 

13. Upward Communication 

^ J- ■ 

14. Satisfaction with Pay 

15. ^ Morale ' 

16. Physical Work! Cond. § Equip 

17. Co-Worker Cooperation 7- . 

18. Operational Effici-ency V-^ ^ 
^19. Workload Balance "V \ 
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Level'^U validities 
mat|ton about the range 



scores were Cperation^T'E^ Morale . -^FiT^lX., Percent 

Special Applicants predicted by 32% -pf^^the^aN^^- most 

valid" score being Opi^^tibf^al SB&ciency/v^-'''"'^ 

Opera-tional Ef £iQlenty\VF^s theTlbst val\d^MAS scor£. J^edfcting all of the 
selected criterion^Tnias'uipes. significantly^ This f^^^?wap followed by. a / 
nurtb.er of others^ Infcluding Work Satisfaction, P^i^^lriance Feedback, Oppor- 
tuitit^ .for ^Promotion/^atisfa^^^ arid Morale. These scores cor- 

related^^s.ignificantly with' five of the six criterion measures. 
', ' . ■ \>- 

• While these results ar,e appropriate when considering^ 
obtained on the Jtptal sample, < they do not provide ini 

of validities., o^tained'^^ l separate states nor dcafe^ validities 

-^ .v^ against some p^thfer in^ortant criteria. The validit)«am|p the Oper- 

;>y^' atipnal Efficiency score in predicting the separate cr^rerion^measures for 
each state are presented in Table 19. In. this table, the only^jiprrelations 
; . of consequence ^or Miss^^^^j^j|g ^ or- Direct Employment Cost "pe'r Placement and 
• ' fndividuals ;pi'aQ^:^d per however, these are two of the highest 

validities in tfi^ table, .^^^^■fci Carolina of the validity coefficients 

• were above 20; fiij^T^ of the validity§»^fficients for Operational 

.. . Efficieacy' wer6 abpye . 30 , bur^me correlation was negatiVe for predicting 
Individual 5 >Piace^^^^^^^ ^n Ohio, the validity coefficients were 

gejierally ill.* t^^^ Individuals Placed per ES Positibn had a validity 

coefficient f6f:".4|^^ pf the higher validity coefficients obtained in the 
study. In Texas, tlie^ Efficiency score correlated ^.bove .20 with 

':Qn iy two o£ t^'6*^cr^ measures, while in Missis'sfppi ) Operational Efficiency 

had iubstanjtl'kd^ i^^^^^ one of the criterion measures. Indivi- 

duals .Placfifi.a'^W ' ^ 

TRese resul^^^* i?^^ Operational Effici^cy fneasui^e had some 

ajipli'cabilit^/ja^^ states, buY there were many exceptions and the results 
should ;bevan^e^^^ cautiously in conjunction with ari other infoifmation 

^ilable.'-^ This is the general recommendation repeated jthroughout the Super- 
|v ylsor)^ Handb^^^ Management Audit Survey which stiated ti^t .all other 

inforrnatic^nvNEtvailabl e sh6uld be conside^gt^jn interpijp^ng the MAS scores^ 
in ti;iew o|:-tf^^^ the current s^udyS^Jhi^^dSfifi^^i™ still 'liolds. 

On the othei^y hl^iiH, the opportunity^uf* afialyzing management procefdures in 
conjunction'^;.wi'th other criterion and control yscor^^jdata offers a significant 
opportifliity for achieving the goals of Employinent Service activities in pro- 
ducing higher effectiveness for ES operations: This canbe illustrated by 
considering one of the criteria, which was not selected for intensive study 
during 'the course of the investigation because of the relatively small Sample 
size, a mea^are of Total Cost per Plaqement. Data were obtained on this 
measure from three states, Missouri, North Xarol. in and Tennessee, with a 
total sample si2:e of 78 offices. The totalHample validity coefficient -for . 
the Operational Efficiency score predict^-ng this c^terioTi was -.45.* The 
results of applying this prediction to the tptal ^^^ 4olJars per placement. *.• 
data fi*om,.the yState of Missouri are presented in Table-'ift^r The' units whi^ch 
scored in the top 25% (^ Operational Efficiency had an ES Dollar^ per Ipace- 
ment average of $57.76, whilfe for those units scoring -in the bottom 25th 
percentile, the average total co^t per placement was $£7:43. These results 
indicated more" than a 50% difference in cost per placoSjent between the upper 
and lower" unit s~~OTi Opera~tioimT"Effi^ 
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Table 19 

efficients 

, Selectfft GriteajjjLon Measures Across Six S£|l;,e*s 



alidity Coefficients of Operational^3^ffici^ty with 
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Table 20 

Average Tot^l ES Dollars per Placement for 
^ Units in the Top 25% and Bottom 25% on 
the MAS,^O^eratipnal Efficiency Score* 
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♦Missouri mean (72.^0) and standard deviation (25.95); 
total sample validity = -.45, n = 78 offices. . 
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for improving and integrating HS performance through'attOTtion to manage- 
ment procedures as assessed by the MAS- s . 

The MAS Validities in Oh io 

4 he highest^ MAS valid'^t^^^^ criteria were 
ound in th^tstatfe p£v'0ifti^^^^^ validities can be found in Table 21.,^ 

These" validities offei|^som^ findings pf th^^ Earlier 

study in Pennsylvania. " • : ... 



For example, on Individuals 
had significant validities, 
had significant Invalidities 




Placed pe^'Ref erral , 11 of t)>e ''i'St MAS'-scc 
In the Pennsylvania study,/. 10 of thevMA 
seven MAS -scores had significant validities ^^^^ 
both states, including Fairness o.f Management , Planning and Adminisitrstt^i^K ^ 
Efficiency, Upward Communication-, Co-'Worker Cooperation, and Operational,;'.^ 
Efficiency. " . " ^ 

On'tho other hand, certain MAS scales did not prove to be quite as effeC- ^ 
tivev4n Ohio. For example. Climate, for Innovation and Downward Communica-T 
tion were among the best MAS scores 'for predicting the performance criteria 
in Pennsylvania, but these two scores were not wfective pjredictors in^Ohio., 
The scores that were identified as being ineffective in. Pennsylvania wejrit 
also ineffective in Ohio, with the exception of FaimesJfcf Management,^, 
which had Useful validities on two of^the six criteria.. ^ 



.' tjiiV^^ the criteria generally followers thje resu;ts obtained 

bn^ the/total sample, .with Percent of Openingis Filled and Individuals Placed 
.pe^iv Referral being the mo^t predictable. Also, as in the total sample, 
^Operational Efficiency was the most consistently valid predictor, correla- 
ting significantly with all six of the selected criteria. 

-fetause of the pattern o^corr^Mitions among the MAS scores and control 
^>6^rc^, the magnitude'of the ysLTidities would change slightly and differen- 
iNs^.trdlly if the effects of the control scdrd^v®^® to he held constant. For' 
example > the ^ validity for Operational Efficiency predicting Indivi^jj ^ ^ 
I>I^aced per ESi.Tosition Would drop from .43-Iq^v.^9, if Workload wer:MKd 
-' constant across, all work groups. The validity for Operational Effi«^cy' 
Anz Percent of Openings Filled woulci increase from .34 to .36, *"if 

Total Unemployed were held constant. However, thef validity 
5n with Pay predicting Percent of Openings Filled would de- 
)^tc(/23, if the effects of total Unemployed were held con- 
lilferorder and second-order partial correlations would generally 
2 4^ s;* but the magnitude would typically be small, 

ficiejtts found in Ohio wer^ obta:j^ed ^^ainst criteria for 
v^rhi'cji^rfec^ed the administration of -Ihe MAS' by three 
l^uTtS'^frofi^ MAS asse^ment of management procedur^es 
fM^lljbonsMps to criteria of ES .performance over an ex- 
o^art^achiteVeml^ which' indica,tes ifi this eiample the impor- 
agenient procedureJ^TO > ^ ^ . . 

obfained'xwillh the MA§\indicated t^hat: 



^ipredij^t 
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There is ample evidence of differences in 
procedures ^within andr|J8tween states^^ 

V # L . ° 
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Table 21 




Validly Xloef ficients of MAS Scores in 
P^retUcting Sej/cted Criteria in Ohio 

between 38 and^65) - 
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^Although the validities were limited in magnitude, there 
.vas^a general tendency for MAS scores to have a high 
percent of significant validiti^ in predicting selected 
criteria of ES performance. 

Criterion unreiiability and control score effects made 
prediction wit^ the MAS difficult. In view of the 
generally consistent results, and the high validities 
obtained in selected states, such as Pennsylvania and 
Ohio, the most important issues deal with if and how ♦ 
MAS information' can be used to*bring about higher 
levels of ES performance. / ^ 

V 

The MAS system, in conjunction with effective criterion 
measures and control * scores, offers an opportiinity to 
initiate sound management procedures throughout the ES 
system. . 




CHAPTER. VI 

REVIEW AND RECOMMENDATIONS FOR FUTURE RESEARCH 

This research* has invol^ved an analysis of extensive amounts of data con- 
cerning Emploj^ment Service operations, Descf i]>tions of management pro- 
^dures were collected from 10„700 employees, and criterion and control 
More m^ures were collected on 451 local ES ojffices. These data wer^ 
collected across six different statfe, during different time periods, so 
the project has cut across ^Y;arious ^onomic and geographical situations. 
The results of the res^f^h are complex and difficult to^summarize. Some 
of the' more important findings are summarized below. 



' Review # 

Liability coefficients were coipputed on fo6r states--Mississippi , North 
Carolina, Tennessee, and Texas--on^which data were available. 

There was some variation from state to sfate, but the reliabilities, gen- 
erally speaking, were limited, i.e. m^^y^corret^tions were t)eloK; . 60. ^ 
The more unreliable the d^ta, the more difficult it' is * to show the impact 
o-f management procedures" Pij, ES^-performance. , TJh^Stunrel lability aF the jper^ 
formance data also makes policg^'decisiohs aiid. higher level management pro- 
cedures mor^ difficult . l>'r' ^ / 



Considerable criterion complexity characterized the results obtained from 
in(|^$ures^**tO: a^^sess' flf^l^^ent Service performance. Six criteria w^^e se- . 
leeted for mor^-extensive study' ^ of their importance %o the Employ- 

ment Service andt tfteir mbcierate interrelationsmipslw indicated some 
'^coni^rgei^e of management procedures in producing^ h,i|h levels bf ES pertur- j 




enrol led in training) measure was not- related to dther ES perfciirm^nce measures^? 
';ind:ic:ating that a conti^nued review o.f these'feinds Qf '"^eiStta^^ 
Alsb, cjilalj.ty of ^he pTacements (higli-wage and Iwg-temi p\atcements) did not 
rfeceive as Qiux^h emphasis im the ES/as it should,^^5Kiace thele^ had 
only iJLimted relationships to t,he other critpria. y y . 

^ ^xt ' ' ' 

^ Workl*^'^.^^^ the^most important control score, 'having sij^jificant influ^ 



on se\r^^ of the criteria. . Tha dif ferent^^il effects of^|farkload on -tWD^^^^^^^^ 

criteriS;^7%^{;iv^^^^ intact ^^on Individuals Platced per-'|^pp)^icant Availabi^f^^j^ 

and 'a ^b3^|l^*\^^^^^^^ on^jIn<^iviclualk^^ pe:f^)ES 'Position-Indicated that 

more rps^Kx'clIf: - policy implication^ of these results. 

'The proportibn and the pr(;^portion ^il^claimants in the 

client population w'br^^^^^^^ have ^ -^egatiive i^ifluencfe^^bn the productivity 
oT ip work groups. 



Control scopes accounted for 52% of/ the A^a«ari^p5^^ 

per Applicant Available criterion. #UI Claimants\^e.r Appllicant^^|^^ble* 
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Workload were the most . important control scores, together accounting for 
30% of the va^riance. Control scores accounted for 46% of th^ variance of 
a second criterion-~Placements per ES Position--with WoTjcload being the 
most substantial predictor, accounting for 26% of the variance by itself.' 
Because of the unreliability of the criterion measures and their dependence 
upon these control scores, little variance remained to show the impact of 
management procedures as i^ssessed by the MAS. ' 

Control scores were factor analyzed to determine the underlying dimen?;tons 
and common characteristics of. the measures. Four factors were found to be ^ 
of importance--Size, Urban-Rural type' of employment, Wol^kload, and Special 
Applicants. This analysis helped to defiT\e the nature of the control var- 
iables that may influence ES opeis^tions at the local office level. 

Mississippi had the highest percent favorable Scores 6n 11 of the 19 MAS 
scoa?6s, and Ohip^jikd the lowest scores on 15 MAS me~asures. Imprpvement on 
the Ohio scores l^puid be expected with differe/it management practices. Across 
the six states, the highest pe'rcent favorable scores were found on" Work Satds- 
faction. Operational Efficiency, and Co-Wofker Qooperation, indicating that 
the ES employees in these states found satisfaction in the work they were 
doing, felt that their units functioned ^Well , and cipperated with eath other 
°iqft-itjgir work.. Low scores were found on' Opportunity for Promotion^ and Satis-- 
■^^'^^^tapn with Pay. 

The control scores dealing^th 'office • charactepistics or situational factors 
describing the setting in which the loc^L office operated were examined to 
determine their imoact on thfe descriptians^qf management procedures as 
' assessed by th^ MA5. • Size was identified as'an important control variable^ 
larger offices had low scores on many^of the MAS variables , ^confirming th^ 
results 'obtained with tl^e crit^on measures. The results of the MAS-cofltrol 
score relationship analys^rs-^^artially supported* the earlier findings in 
Pennsylvania. ' ' ' * ^ 

I The criteria were predicted witft significant validrties by a high percent^ of 
^the scores, between 32 ahd 79%, aiyiiough the magnitude oF^the correla- 
tions was limited. The most predictable criterion waS Individuals "Placed 

■ per Referral. This was followed closeajU^y Pewint of Openingf Fill^, which 
was^ significantly predicted by 74% pf 'the MAS scores. Operational Effi- 
ciency was. the most valid MAS score ar^j^^ having significant validities on 
all 'six of. the sheeted criteria.. Opportunity for Promotion, Morale, Per- 
forman9e FeedbackXand Satisfaction with Pay were also gene-r^ly good pre- ^ 

. dictors, having sigjnifican't validities on five of the six ^ritWia/^ Thete 
were marked "differehces for the various MAS scores across states , with .the 
performance criterik in the State of Ohio being the most predictable. 

■ The validities in the "State of Ohio were the highest overall, .corroborating . 
many of . the~earl±ei^ Pennsylvanxa~ fitidirigs . S^en MAS- scores had significant 

^ validities on both states in predicting Individuals Placed per R^erral . 

" The predictability, of the criteria generally ' followed the: results obtained , 
on the^ total sample, with Percent of Opei^^ings Filled ahd Placements per 
Referral being the most 'predictable. Operational Efficiency was the most % 

• consistently valid predictor, having significant validitLfes on all six / ^ 
criteria. The validity coefficients found iivOhia wansjpbtained against 

— crtteri^r^oi^a^rZ^DTrt^^ 




by three months. The results from MAS assessments of management pro- 
cedures in predicting different criteria of ES performance over such an 
extended time period indicated the importance of the research in measuring 
relevant management practices which, could be used to foster more effective , 
ES<iL-aperations. ^ ' - 

On t<he basis of this research, the important question of the validity of 
the MAS in predicting ES performance criteria has noct been definitively 
answered. While a high*percentage of significant validities were obtained, 
the generality of the results in each of the participating states was limited, 
partially because of the unreliability of the criteria and the influence of 

he control scores A more inclusive question would make the following 
imjimry: Do the results indicate that the MAS can be used with other manage- 
ment information to strengthen management practices and improve ES perfor- 
inance? The answer to this question is partially dependent upon future 
research and ^upon interpretations of the current results. Sinr^there were 
numerous iodications of significant empirical relationships between MAS 
assessment^ of management practices md ES performance criteria, the answer to 
this question would be positive, if an effective system of recommended (proce- 
djares for using MAS results with, all other available information were avail- 
able. Thi3 . issue and the findings . from the current study lead to the, fol- 
•l-owing recommendations for future research. " v*/ 




Recommen'dations f^saff^utuxe Research 



1) There is not a set of recommended procedures for using MAS Z-'t^ 
an integrated strategy of organizational development (OD) that makes full 
use .pf other available criterion 'and control score information . Since 
MAS scores are disseminated throughout an organization from upper levels 
down to- lower level work groups, the combined system should have, broad 
applicability. Thus, one of the pbssibili^ies for^future research would 

be to design, impj^^ment, and evaluate OD sftrategiesij including use of - ■* 
criterid'n and control score information, .MAS result^-- aKi^Sither- OD techniques 

so that evaluation evidence as well as ;a set of recommended procedures would 

be available to guide future implement^ion and dissemination efforts. Suc)\ 
•a study would involve the collection of criterion-control score .information 
Id MAS results at the beginning of the study, followed by, various kinds of 
intervention efforts and repeated measures obtained on criterion-control 
score information and MAS scores. The intervention procedures would include 
information dissemination techimiuesw-ipmployee participation, dfevelopmeat 
of action plans, continuous feOTback T'oops , etc. This phase would then / 
be followed by another phase where other organizational components j^^^npt ^ 
studied initially except as a control group, would implement^the i&^^,j^t- 
ing recommended procedures on, their own without the support and int^eryen'^ 
"tibn of the invoi'tigatoYs . At the conclusion of this second pha^e, a^ddi^ ' 
tional criterion and ctorftrol score dat^ would be collected^^to.j^ all 
procedures further, . -ir^ ' **"^^ - ' 

2) As mentioned- in the first recommendation for future reserach, a manual 
of recommended procedures i^ needed which Wduld describe how MAS results 

cam 'be used effectively in conjunction with other information. Sincei the , ^ 
MAS ha^ been administered to such large samples of ES persoTr^el (dver 20,000 
employees), designing such a- man^^ of reifpmmended procedures .on the bdfeis 




of current experiences of states where the MAS has been administered, 
and upon all the available research findings^ without doing a long-term, 
organizational development study would be a reasonable task. The manual 
could then be modifi>ed as a result of future experience, using these 'pvQ- 
' "cedures. Obviously, this strategy would be much more economical, but 
Jess thorough, than an extensive OD study. 

3) As described in^ the early chapters of this report^ the MAS had its 
origin as ^ supplement to conventional auditing techniques, so that a 
wide variety of management procedures could be economically examined and 
feedback pr-qvided at^all levels of the organization. In light of the 'cur- 
rent research findings, the MAS could be combined in an intjegrated fashion 
with an automated audit report analysis system which would Consider a 
limited-number o-f criterion and control scores such as Individuals Placed 
perAp^icant Available, Individuals Placed per ES Position, Workload, 

and UI Claimants per Applicant Available, and the data analyzed via a 
computer-generated report for auditing and management personnel. The 
system could be highly efficient; that is, the report written within two 
.weeks after completion of the survey. The output would include an analysi 
of management procedures and the criterion measures and theiu interrela- 
tionships to identify problem areas and opportimlr1UJ»^ for improvement in^ 
local offices. Such a system would go some distance toward integrating 
criterion performance, control score^, and management procedures into 
a sophisticated system *to '^foster achievement of ES goal^s. 

4) '•^The Management Audit Survey was constructed to be a general-purpose 
instrument which woulc^ be appropriate for a wide range of organizations 
arra functions within large organizations.. The results of the current 
s^dy supported many of the findings in the "earlier study on Pennsylvania 
that certain key scores ar^more important than others for ES operations, 
e.g.. Operational Efficiency, Performance Feedback^ Morale, Downward. Com- 

• munication, etc. '^^'Fox these important score areas and others, redesigning 
^^uestions a.^d sc&t^ef' areas within the MAS to be specifically applicable to 
managemiz^t^/frt^cedures of ES operations would be relatively Efficient and 
econonyfcal. The rewriting of items would incorporate vocabulary andr terms 
that vould integrate manajgejnfent behaviors land procedures as part of the 
MAS sffetem. The new foxm would thus^^^e specifically designed for ES 
operations, would be' economical , and could be used periodically U) assess 
the Relevant management behaviors. ^ 

'J - • . - 

5) A3 stated previously, one of the. key issues is how to make best use 
of the results obtained in the current research to bring about higher . 
levels of ES performance; The recommended research numbered 2), 3)., and 
4) above could be integrated.-to acfiieye,..S:Uchr a goal. Such a combined 
system offers a number of advantages: - A) definition ^nd^quantification 
of ES goals and performance indica^^[j^; of ;management pro-' 
cedures which underly the|pchievement;. of such goals; and'CJ^ a means of 
integrating the different sets of measures of\ management assessments, 
criterion measures , and control scores into an^ integrated' system. 

6) The research ^-indicated that control scores have an important effect 

on ES peffo.rmance and yet 'there were strong indications of the instability 
of the criterion-control score ra^ationships . Thus the results obtained' 
in the present study as. well as those obtained by otjier invesitigators 
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•indicate that the important control score effects should be cross-validated 
•over time, across geographical areas, and across units of analyses (i.e., 
local offic/5s, SMSA's, and states) . Without morev definitive evidence about 
the magnitiAde of control score effects to determine the generality of J^he 
results and their implications, management cannot evaluate the actuaju^ 
•nature or extent of control score effects. 

7) The research revved that there were not only reliability problems in 
the criterion data bm also quality control limita*tions as a number of 
'local offices hid scores which were simpl>^ impossible on the measures in- 
volved. Thus, an opportunity for future resea^rch is the dey.eropment. of 

a simple, but ^ef fective , quality con^ol system whiph would bQr entirely- 
computer-g^Verated for.a,naiyziV(g the jacGuracy of E^ARS performance data. 
Such a :comput€ri2ed;^§^st^ magnitude of the 

results ^Db;ts^i^^;M^ as-,we^4'-a^^^ relationships between- 

numerate r,,fS0^ other Criteria, etc. -§uch 

a system wolutcj/ cbritribu^e to better quality of ESARS data and in itself 
could be a criterion measure of local office and state p^^^rformance. 

8) The remits indicated that Workload^as an important variable in 
,»influencing ES, performance. Furthermore, the variable had differential 

impact on different kinds of criteria--Individuals Placed per Applicant 
Available and Individuals Placed per ES Position. The importance of th'ese 
findings suggests that a nation-wide studyjshould be conducted using avails 
able data to examine the standing oS each state on the Workload variable. 
With this information, the significance of different levels of Workload 
could be evaluated in conjunction with national priorities and goals about 
the relative importance of these two criteria. Such information would be 
helpful in directing resource expenditures to accomplish national goal^. 
Such a study should involve not only state-level data but- data at Ipcal 
office level on a ssanpling basis. In this manher, an understanding 
could be obtained Q.f /the da'ta^ aggregation process as different unit's of 
analysis are used. The study should consider regional and state office 
staff separately in looking at the outcome between the criteria 'and the 
Workload measure- and* the results integrated to make overall conclusions 
about staffing priorities. 



Research on ES operations, performance, extent of' service, client groups, 
/ ^ etc., in different business environments, with varied p^itical and t^ocial 
P2l.ii;y considerations will continue to be extremely compTex. The results 
MAS system provide an opportunity for improvement in ES performance, 
characteristics of the MAS represent a level of considerable sophis- 
^n dn the m'easur,§me'nt of management procedures and .organij^zational 
climafe. Yet, as subs^arTTrai iinformation about the ES continufe^to^ccum- 
ulate, this leads to further^roblems of implementation and issu^^t?p;pf^;lic.y . 
and direction. ..l^ie tFfaraPctei^stics and furth^ growth of such an open sys^j^ 
remain as a continued challenge for future r*^Warch and for ES management 
and employees. j ' V 
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and Ofhcr Variable! Across Individuals (N=12,131) 



No. 



'/" Description 
• " /. 
.1. Fairness of Management 

: 2. 'Delegation of Authority , SO 

■ ^ 3. Supervisory Effectiveness : 58 

4. Planning .fi, Admin. Effic. 43 

- * S. CI iinate fot Innovation \ 

f). Wk Satisfaction 34 

f;. 7, '. Training Effectiveness '48 

Perfornance Feedback 51 

9. EEO Women'] - 32 

I ■ \o. EEO Minority .Program ' 23 

f' 11. Opportunity for Prjmotions _ 50 

12. Downward Comunication 59 

13. Upward Communication " 62 

14. Satisfaction with Pay • 24 

■ is. Morale • ^ 
1^. Physical' Working Conditions 

' 17.. Co-Worker Cooperation 
, 'ill. /♦operational Efficiency 

• . .191' Kogkljiad Balance 

, 20.' (iAnderstaffing Responses 

'(21. I OverStaf^fing Responses; 

' '22, Admin/Prof = 1 other = 2 

23. Minority - 1 other = 2 

9 8 24. Supervisor « 1 other » 2 

25. Organization Level , 



1 2 3 4 5 J ,7 I 8 ,9 10 11 12 13 14 15 16 17 18 ^' 20 '21 22,23 24 25 ;. 



50 



/ 



20 22 13' 




V 



17 15 50 ^"' 
144 46 29 16 -- ^ 
61 53 p, 20 47 -- ^ 
52 S3 30 ^ 54 66 ' 
21. 15 11 08 36 23 24 
50 46 25 18 56 55 S7 28 - 
27 , 19 13 07 29 26 27 19 35 ■• 
41 ^6 '21 17j- 31 45 42 11" 54 20 ■• 
43 33 19 23 26 42 36 05 ^ 48 20 62 

18 09x 01-03 14 IS 14 20 20 13 10 03 - . 
-1&^1)2 '02 05 -Or-08^07 -20 -12 -12.-05 02 -83 - 
-13 -13 -0M6' -08 -13 -12' 06 -U 00 -12 -14 -09 -36 
,i"-a7^il -12 -09 .02 -OS 00 -08 -09 -07 -05 07 •O? 00 
02 '-O^' 11, 33 0! Ol/OMO* 05 05 03 07 00 00 ^0-09, 



-03 -08 



■n -09^5 y 




5 -M il3 -OS -13 -OS -08 -06 06 -09, 05 27 -OS 



11 00 04 0*3^-08 -09 -10 -it -17 -08-11.-07 08 -04' 00 -12 18 



Q N'ote: all decimals have been omitted. 



